CT-Net: Channel Tensorization Network for Video Classification

3D convolution is powerful for video classification but often computationally expensive , recent studies mainly focus on decomposing it on spatial-temporal and/or channel dimensions . Unfortunately , most approaches fail to achieve a preferable balance between convolutional efficiency and feature-interaction sufficiency . For this reason , we propose a concise and novel Channel Tensorization Network ( CTNet ) , by treating the channel dimension of input feature as a multiplication of K sub-dimensions . On one hand , it naturally factorizes convolution in a multiple dimension way , leading to a light computation burden . On the other hand , it can effectively enhance feature interaction from different channels , and progressively enlarge the 3D receptive field of such interaction to boost classification accuracy . Furthermore , we equip our CT-Module with a Tensor Excitation ( TE ) mechanism . It can learn to exploit spatial , temporal and channel attention in a high-dimensional manner , to improve the cooperative power of all the feature dimensions in our CT-Module . Finally , we flexibly adapt ResNet as our CT-Net . Extensive experiments are conducted on several challenging video benchmarks , e.g. , Kinetics-400 , Something-Something V1 and V2 . Our CT-Net outperforms a number of recent SOTA approaches , in terms of accuracy and/or efficiency . 1 INTRODUCTION . 3D convolution has been widely used to learn spatial-temporal representation for video classification ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ) . However , over parameterization often makes it computationally expensive and hard to train . To alleviate such difficulty , recent studies mainly focus on decomposing 3D convolution ( Tran et al. , 2018 ; 2019 ) . One popular approach is spatial-temporal factorization ( Qiu et al. , 2017 ; Tran et al. , 2018 ; Xie et al. , 2018 ) , which can reduce overfitting by replacing 3D convolution with 2D spatial convolution and 1D temporal convolution . But it still introduces unnecessary computation burden , since both spatial convolution and temporal convolution are performed over all the feature channels . To further decrease such computation cost , channel separation has been recently developed via operating 3D convolution in the depth-wise manner ( Tran et al. , 2019 ) . However , it inevitably loses accuracy due to the lack of feature interaction between different channels . For compensation , it has to introduce point-wise convolution to preserve interaction with extra computation . So there is a natural question : How to construct effective 3D convolution to achieve a preferable trade-off between efficiency and accuracy for video classification ? ∗Equally-contributed first authors ( { kc.li , yl.wang } @ siat.ac.cn , xianhangli @ knights.ucf.edu ) †Corresponding author ( yu.qiao @ siat.ac.cn ) This paper attempts to address this question by investigating two design principles . ( 1 ) Convolutional Efficiency . As shown in Table 1 , current designs of spatial-temporal convolution mainly focus on decomposition from either spatial-temporal ( Tran et al. , 2018 ) or channel dimension ( Tran et al. , 2019 ) . To enhance convolutional efficiency , we consider decomposing convolution in a higher dimension with a novel representation of feature tensor . ( 2 ) Feature-Interaction Sufficiency . Table 1 clearly shows that , for current decomposition approaches ( Tran et al. , 2018 ; 2019 ) , feature interaction only contains one or two of spatial , temporal and channel dimensions at each sub-operation . Such a partial interaction manner would reduce classification accuracy . On one hand , it decreases the discriminative power of video representation , due to the lack of joint learning on all the dimensions . On the other hand , it restricts feature interaction in a limited receptive field , which ignores rich context from a larger 3D region . Hence , to boost classification accuracy , each sub-operation should achieve feature interaction on all the dimensions , and the receptive field of such interaction should be progressively enlarged as the number of sub-operations increases . Based on these desirable principles , we design a novel and concise Channel Tensorization Module ( CT-Module ) . Specifically , we propose to tensorize the channel dimension of input feature as a multiplication of K sub-dimensions , i.e. , C = C1 × C2 × · · · × CK . Via performing spatial/temporal separable convolution along each sub-dimension , we can effectively achieve convolutional efficiency and feature-interaction sufficiency . For better understanding , we use the case of K = 2 as a simple illustration in Figure 1 . First , we tensorize the input channel into C = C1×C2 . Naturally , we separate the convolution into distinct ones along each sub-dimension , e.g. , for the 1st sub-dimension , we apply our spatial-temporal tensor separable convolution with the size C1×1×t×h×w , which allows us to achieve convolutional efficiency on all the spatial , temporal and channel dimensions . After that , we sequentially perform the tensor separable convolution sub-dimension by sub-dimension . As a result , we can progressively achieve feature interaction on all the channels , and enlarge the spatialtemporal receptive field . For example , after operating 1st tensor separable convolution on the 1st sub-dimension , C1 channels interact , and 3D receptive field of such interaction is 3 × 3 × 3 . Via further operating 2nd tensor separable convolution on the 2nd sub-dimension , all C1 × C2 = C channels have feature interaction , and 3D receptive field of such interaction becomes 5×5×5 . This clearly satisfies our principle of feature-interaction sufficiency . We summarize our contributions in the following . First , we design a novel Channel Tensorized Module ( CT-Module ) , which can achieve convolutional efficiency and feature-interaction sufficiency , via progressively performing spatial/temporal tensor separable convolution along each sub-dimension of the tensorized channel . Second , we equip CT-Module with a distinct Tensor Excitation ( TE ) mechanism , which can further activate the video features of each sub-operation by spatial , temporal and channel attention in a tensor-wise manner . Subsequently , we apply this full module in a residual block , and flexibly adopt 2D ResNet as our Channel Tensorized Network ( CT-Net ) . In this case , we can gradually enhance feature interaction from a broader 3D receptive field , and learn the key spatial-temporal representation with light computation . Finally , we conduct extensive experiments on a number of popular and challenging benchmarks , e.g. , Kinetics ( Carreira & Zisserman , 2017 ) , Something-Something V1 and V2 ( Goyal et al. , 2017b ) . Our CT-Net outperforms the state-of-the-art methods in terms of classification accuracy and/or computation cost . 2 RELATED WORKS . 2D CNN for video classification . 2D CNN is a straightforward but useful method for video classification ( Karpathy et al. , 2014 ; Simonyan & Zisserman , 2014 ; Wang et al. , 2016 ; Liu et al. , 2020 ; Jiang et al. , 2019 ) . For example , Two-stream methods ( Simonyan & Zisserman , 2014 ) learn video representations by fusing the features from RGB and optical flow respectively . Instead of sampling a single RGB frame , TSN ( Wang et al. , 2016 ) proposes a sparse temporal sampling strategy to learn video representations . To further improve accuracy , TSM ( Lin et al. , 2019 ) proposes a zeroparameter temporal shift module to exchange information with adjacent frames . However , these methods may lack the capacity of learning spatial-temporal interaction comprehensively , which often reduces their discriminative power to recognize complex human actions . 3D CNN for video classification . 3D CNN has been widely used to learn a rich spatial-temporal context better ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Feichtenhofer et al. , 2019 ; Sudhakaran et al. , 2020 ; Feichtenhofer , 2020 ) . However , it introduces a lot of parameters , which leads to a difficult optimization problem and large computational load . To resolve this issue , I3D ( Carreira & Zisserman , 2017 ) inflates all the 2D convolution kernels pre-trained on ImageNet , which is helpful for optimizing . Other works also try to factorize 3D convolution kernel to reduce complexity , such as P3D ( Qiu et al. , 2017 ) and R ( 2+1 ) D ( Tran et al. , 2018 ) . Recently , CSN ( Tran et al. , 2019 ) operates 3D convolution in the depth-wise manner . Nevertheless , all these methods still do not achieve a good trade-off between accuracy and efficiency . To tackle this challenge , we propose CT-Net which learns on spatial-temporal and channel dimensions jointly with lower computation than previous methods . 3 METHODS . In this section , we describe our Channel Tensorization Network ( CT-Net ) in detail . First , we formally introduce our CT-Module in a generic manner . Second , we design a Tensor Excitation ( TE ) mechanism to enhance CT-Module . Finally , we flexibly adapt ResNet as our CT-Net to achieve a preferable trade-off between accuracy and efficiency for video classification . 3.1 CHANNEL TENSORIZATION MODULE . As discussed in the introduction , the previous approaches have problems in convolutional efficiency or feature-interaction sufficiency . To tackle such a problem , we introduce a generic Channel Tensorization Module ( CT-Module ) , by treating the channel dimension of input feature as a multiplication of K sub-dimensions , i.e. , C = C1 × C2 × · · · × CK . Naturally , this tensor representation allows to tensorize the kernel size of convolution TConv ( ) as a multiplication of K sub-dimensions , too . To simplify the notation , the channel dimension of the output is omitted by default . The output Xout can be calculated as follows : Xout = TConv ( Xin , W C1×C2×···×CK×t×h×w ) ( 1 ) where Xin and W denote the tensorized input and kernel respectively . However , such an operation requires large computation , so we introduce the tensor separable convolution to alleviate the issue . Tensor Separable Convolution . We propose to factorize TConv ( ) along K channel sub-dimensions . Specifically , we decompose TConv ( ) as K tensor separable convolutions TSConv ( ) , and apply TSConv ( ) sub-dimension by sub-dimension as follows : Xk = TSConv ( Xk−1 , W 1×···×Ck×···×1×t×h×w ) ( 2 ) where X0 = Xin and Xout = XK . On one hand , the kernel size of the kth TSConv ( ) is ( 1× · · · × Ck × · · · × 1× t× h× w ) . It illustrates that only Ck channels interact in the kth suboperation , which leads to convolution efficiency . On the other hand , as we stack the TSConv ( ) , each convolution performs on the output features of the previous convolution . Therefore , the spatialtemporal receptive field is enlarged . Besides , interactions first occur in C1 channels , second in C1×C2 channels and so on . Finally , C1×C2×· · ·×CK = C channels can progressively interact . This clearly satisfies our principle of feature-interaction sufficiency . Spatial-Temporal Tensor Separable Convolution . To further improve convolution efficiency , we factorize the 3D TSConv ( ) into 2D spatial TSConv ( ) and 1D temporal TSConv ( ) . Thus , we can obtain the output features XSk and X T k as follows : XSk = S-TSConv ( Xk−1 , W 1×···×Ck×···×1×1×h×w ) ( 3 ) XTk = T-TSConv ( Xk−1 , W 1×···×Ck×···×1×t×1×1 ) ( 4 ) where S-TSConv ( ) and T-TSConv ( ) represent spatial and temporal tensor separable convolution respectively . Finally , we attempt to aggregate spatial and temporal convolution . There are various connection types of spatial and temporal tensor separable convolution , e.g. , parallel and serial types . According to the results of the experiments in Section 4 , we utilize the parallel method , which illustrates that we sum the spatial feature XSk and temporal feature X T k : Xk = X S k +X T k ( 5 )

Summary: This manuscript proposes a novel convolutional operation for learning representations from video data. By decomposing the channel dimension into sub dimensions in the typically 4D video data (Time, Channel, Width, Height), one defines spatial-temporal separable convolution for each sub-dimension. This could improve the representation learning in term of efficiency and modeling quality.