Spatiotemporal Joint Filter Decomposition in 3D Convolutional Neural Networks

In this paper , we introduce spatiotemporal joint filter decomposition to decouple spatial and temporal learning , while preserving spatiotemporal dependency in a video . A 3D convolutional filter is now jointly decomposed over a set of spatial and temporal filter atoms respectively . In this way , a 3D convolutional layer becomes three : a temporal atom layer , a spatial atom layer , and a joint coefficient layer , all three remaining convolutional . One obvious arithmetic manipulation allowed in our joint decomposition is to swap spatial or temporal atoms with a set of atoms that have the same number but different sizes , while keeping the remaining unchanged . For example , as shown later , we can now achieve tempo-invariance by simply dilating temporal atoms only . To illustrate this useful atom-swapping property , we further demonstrate how such a decomposition permits the direct learning of 3D CNNs with full-size videos through iterations of two consecutive sub-stages of learning : In the temporal stage , full-temporal downsampled-spatial data are used to learn temporal atoms and joint coefficients while fixing spatial atoms . In the spatial stage , full-spatial downsampled-temporal data are used for spatial atoms and joint coefficients while fixing temporal atoms . We show empirically on multiple action recognition datasets that , the decoupled spatiotemporal learning significantly reduces the model memory footprints , and allows deep 3D CNNs to model high-spatial long-temporal dependency with limited computational resources while delivering comparable performance . 1 Introduction . Convolutional Neural Networks ( CNNs ) have been used intensively in the field of video understanding . Particularly , networks with 3D convolutional layers capture spatiotemporal correlation and achieve great success in applications like action recognition [ 1 , 30 ] . However , joint spatiotemporal modeling of videos usually requires significant training time , computation , and memory , which becomes one of the main obstacles in video understanding . In practice , we almost always need to first downsample video data spatially or temporally or both to meet real-world constraints of computational resources and training time . For instance , a 64-frame video with the spatial resolution 224×224 is downsampled 2 times temporally for training Non-Local Networks [ 30 ] , and 8 times temporally for the R ( 2+1 ) D models [ 26 ] . As in most cases , for spatiotemporal modeling of high-resolution videos , 3D CNNs can not be fit into most modern GPUs due to the huge model memory footprints . However , it has been observed that the full spatial and temporal resolution are essential to achieve superior performance in many video understanding tasks [ 24 , 27 , 30 ] . In this paper , we propose spatiotemporal joint filter decomposition to decouple spatial and temporal learning while preserving spatiotemporal dependency in a video . As shown in Figure 1 , a 3D convolutional filter is jointly decomposed over a group of spatial atoms and temporal atoms . The two groups of atoms can reconstruct the 3D filter together with the joint coefficients through tensor multiplication . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A single 3D convolutional layer is now divided into three convolutional layers : a temporal-atom layer to focus on time , a spatial-atom layer for space , and a joint coefficient layer to model the spatiotemporal dependency . Thus , spatial and temporal atoms can now be optimized separately . Different from methods that decorrelate the spatial and temporal modeling , the proposed decomposition can still capture spatiotemporal correlations in the joint coefficients . Moreover , our approach can also significantly reduce model parameters and computation complexity by limiting the number of atoms . One obvious arithmetic property of our joint decomposition is to allow spatial or temporal atoms to be swapped with a set of atoms with the same number but different sizes while keeping the remaining unchanged . For example , we can now achieve tempo-invariance by simply dilating temporal atoms only . To exploit this useful atom-swapping property , we further show how the proposed spatiotemporal jointly decomposed convolutional filter ( STDCF ) permits the direct learning of 3D CNNs with full-size videos . We start with a pedagogical learning decoupling to train a 3D CNN iteratively with full-temporal downsampled-spatial data first , and then with full-spatial downsampled-temporal data . Note that , such simple-minded decoupled learning is less capable of modeling spatiotemporal dependency . A regular 3D CNN model trained from the two-stage strategy above may capture rich temporal features in the first stage , however , such features will be severely degraded by the downsampled-temporal data in the second stage . Exploiting the atom-swapping property of the proposed spatiotemporal joint decomposition , we can instead decouple spatial and temporal learning into iterations of two consecutive sub-stages of learning : temporal-focus stage ( stage-t ) and spatial-focus stage ( stage-s ) . In stage-t , temporal atoms and joint coefficients are learned from full-temporal downsampled-spatial videos by keeping spatial atoms fixed , while in stage-s , spatial atoms and coefficients are updated from full-spatial downsampled-temporal data by keeping temporal atoms fixed . Note that , in stage-s , spatial atoms are learned from full spatial-resolution , while temporal atoms are frozen to prevent degradation from temporal down-sampling , similar for atoms update in stage-t . The proposed two-stage 3D CNN training strategy can model a full temporal and spatial resolution video only from downsampled data while preserving spatiotemporal dependency . Empirically , we show on multiple action recognition datasets , e.g. , KTH [ 19 ] , Kinetics-400 [ 1 ] and Something-Somethingv1 [ 10 ] , that , the STDCF model trained with the proposed strategy significantly reduces memory usage , while producing comparable results with the state-of-the-art models trained with full-size videos . 2 Method . In this section , we start by introducing the proposed spatiotemporal jointly decomposed convolutional filter , STDCF , along with its two properties of capturing spatiotemporal dependency and enabling atom swapping . Then , we decouple spatiotemporal learning into iterative two sub-stage learning . Next , we introduce a simple but effective design , enabled by the atom swapping property , to allow the model to learn from videos with different spatial and temporal resolutions in different sub-stages . 2.1 Spatiotemporal Joint Filter Decomposition . A regular spatiotemporal convolutional filter in deep video models consists of a group of 3D tensors with shape t × k × k for capturing local joint dependency on space and time , where t and k are temporal and spatial filter sizes . The 3D structure not only leads to a dramatic increase in parameters compared to 2D CNNs , but also couples the spatial and temporal learning as each tensor attends to both dimensions simultaneously . To decouple spatial and temporal learning , as well as to reduce model parameters , we propose to jointly decompose a 3D filter over spatial and temporal atoms , as shown in Figure 1 . Our idea originates from spatial filter decomposition as proposed in [ 17 ] , and we extend it to 3D filters by taking the time dimension into consideration . As proposed in [ 17 ] , a 2D spatial filter W ∈ RCout×Cin×k×k ( Cout , Cin are input , output channels , and k is spatial kernel size ) , can be decomposed as a linear combination of M spatial atoms { ψi } Mi=1 , ψi ∈ Rk×k , with coefficients α ∈ RM×Cout×Cin . The spatial convolution then becomes two : A spatial-atom convolution with ψi , a 1 × 1 convolution with joint coefficients α . This decomposition not only reduces the parameters and computation cost but also imposes low-rank filter structures . Then , for a 3D spatiotemporal filter W ∈ RCout×Cin×t×k×k ( t is the temporal kernel size ) , as shown in Figure 1 , we decompose it over spatial atoms Ψ = { ψi } Mi=1 and temporal atoms Φ = { φj } Nj=1 , φj ∈ Rt , with joint coefficients α ∈ RCout×Cin×M×N as W = ∑M i=1 ∑N j=1 α i , jψiφj . In this way , the 3D convolution becomes three , Spatial-atom convolution : Zi ( λ′ , t ) = ∑ u′ I ( λ′ , t , u+ u′ ) ψi ( u ′ ) , Temporal-atom convolution : Yij ( λ′ ) = ∑ t′ φj ( t ′ ) Zi ( λ ′ , t+ t′ ) , 1× 1 joint coefficient convolution : J ( λ , t , u ) = ∑ λ′ ∑ i , j αi , j ( λ′ , λ ) Yij ( λ ′ ) , ( 1 ) where I , J denotes the input and output , u ∈ R2 , t ∈ R , λ , λ′ are spatial and temporal indices , λ , λ′ are output , input channel indices , and Zi , Yij are intermediate outputs of spatial-atom layer and temporal-atom layer . 2.1.1 Spatiotemporal Dependency . Note that a popular alternative way of 3D filter decomposition is to apply rank-1 decomposition to obtain a 2D spatial filter and a 1D temporal filter , as proposed in [ 18 , 26 , 37 ] . Specifically , a 3D filter W ∈ RCin×Cout×t×k×k is decomposed intoWs ∈ RCin×Cout×1×k×k andWt ∈ RCout×Cout×t×1×1 as W = Ws ×Wk . In this way , 3D convolution becomes two : spatial sub-convolution with Ws , and temporal sub-convolution with Wt . However , such a rank-1 decomposition neglects joint spatiotemporal dependency . The spatial subconvolution only focuses on spatial modeling , while the temporal sub-convolution merely attends to temporal modeling . The spatial and temporal features are extracted independently by omitting spatiotemporal correlation . Instead , in our STDCF , joint coefficients α encode spatiotemporal dependency while keeping learning in both dimensions independent . We derive below that STDCF has more capacity than rank-1 decomposition . Recall our joint decomposition of 3D filter W , W ( λ , λ′ ) = ∑ i , j αi , j ( λ , λ′ ) ψi ( u ) φj ( t ) , α i , j ( λ , λ′ ) ∈ R , ( 2 ) λ ∈ [ C ] , λ′ ∈ [ C ′ ] , i ∈ [ M ] , j ∈ [ N ] , and αi , j ( λ , λ′ ) are freely trainable . For each fixed pair of ( λ , λ ′ ) , { αi , j ( λ , λ′ ) } i , j is a M -by-N matrix , and generally is full rank . In the rank-1 decomposition , spatial filters are Ws = W ( λ ′′ , λ′ ) ( u ) , temporal filters are Wt = W ( λ , λ ′′ ) ( t ) , λ′′ ∈ [ C ′′ ] . We can write W ( λ ′′ , λ′ ) s and W ( λ , λ′′ ) t as combination of atoms ψi and φj respectively , W ( λ ′′ , λ′ ) s = ∑ i µi ( λ′′ , λ′ ) ψi , W ( λ , λ′′ ) t = ∑ j νj ( λ , λ′′ ) φj . Tracking the degree of freedom reveals that rank-1 decomposition is more restrictive than our STDCF . Let C = C ′ = C ′′ , α in STDCF has MNC2 many variables if all free . In comparison , µ has MC2 many variables , and ν has NC2 many , thus the rank-1 decomposed convolution formulation has only ( M +N ) C2 many free variables in total . This quantifies the loss of expressiveness from STDCF to rank-1 decomposition . A more comprehensive analysis is provided in Appendix A . To empirically illustrate STDCF is more expressive than rank-1 decomposition , we design a toy example on the translate-rotate-MNIST ( TR-MNIST ) dataset . It consists of short clips of moving digits , which contain complex spatiotemporal correlations . Adopting a 3-layer 3D CNN , we conduct video reconstruction experiments and compare the qualitative and quantitative results between STDCF and rank-1 decomposition . Details of experimental settings are provided in Appendix B . As shown in Figure 2 , given a sequence of moving-rotating digits , STDCF reconstructs in better quality than rank-1 . As for qualitative results , we measure mean PSNR on the test set . STDCF achieves a mean PSNR of 24.60 , whereas the result of rank-1 decomposition is 21.50 , indicating STDCF possesses more capacity to capture spatiotemporal correlations .

Summary: This work studies the problem of decomposing 3D convolutions. Previous works only consider the spatial and temporal information separately, e.g., E3D, P3D, without jointly considering the spatiotemporal information. Therefore, this work decompose the 3D convolution into a temporal atom layer, a spatial atom layer, and a joint coefficient layer. Besides, by the atom-swapping property, two training strategies are proposed to effectively train the proposed network. Experimental results show the proposed approach outperforms the baselines in terms of memory usage, while the performance remains comparable.