Spatiotemporal Joint Filter Decomposition in 3D Convolutional Neural Networks

In this paper , we introduce spatiotemporal joint filter decomposition to decouple spatial and temporal learning , while preserving spatiotemporal dependency in a video . A 3D convolutional filter is now jointly decomposed over a set of spatial and temporal filter atoms respectively . In this way , a 3D convolutional layer becomes three : a temporal atom layer , a spatial atom layer , and a joint coefficient layer , all three remaining convolutional . One obvious arithmetic manipulation allowed in our joint decomposition is to swap spatial or temporal atoms with a set of atoms that have the same number but different sizes , while keeping the remaining unchanged . For example , as shown later , we can now achieve tempo-invariance by simply dilating temporal atoms only . To illustrate this useful atom-swapping property , we further demonstrate how such a decomposition permits the direct learning of 3D CNNs with full-size videos through iterations of two consecutive sub-stages of learning : In the temporal stage , full-temporal downsampled-spatial data are used to learn temporal atoms and joint coefficients while fixing spatial atoms . In the spatial stage , full-spatial downsampled-temporal data are used for spatial atoms and joint coefficients while fixing temporal atoms . We show empirically on multiple action recognition datasets that , the decoupled spatiotemporal learning significantly reduces the model memory footprints , and allows deep 3D CNNs to model high-spatial long-temporal dependency with limited computational resources while delivering comparable performance . 1 Introduction . Convolutional Neural Networks ( CNNs ) have been used intensively in the field of video understanding . Particularly , networks with 3D convolutional layers capture spatiotemporal correlation and achieve great success in applications like action recognition [ 1 , 30 ] . However , joint spatiotemporal modeling of videos usually requires significant training time , computation , and memory , which becomes one of the main obstacles in video understanding . In practice , we almost always need to first downsample video data spatially or temporally or both to meet real-world constraints of computational resources and training time . For instance , a 64-frame video with the spatial resolution 224×224 is downsampled 2 times temporally for training Non-Local Networks [ 30 ] , and 8 times temporally for the R ( 2+1 ) D models [ 26 ] . As in most cases , for spatiotemporal modeling of high-resolution videos , 3D CNNs can not be fit into most modern GPUs due to the huge model memory footprints . However , it has been observed that the full spatial and temporal resolution are essential to achieve superior performance in many video understanding tasks [ 24 , 27 , 30 ] . In this paper , we propose spatiotemporal joint filter decomposition to decouple spatial and temporal learning while preserving spatiotemporal dependency in a video . As shown in Figure 1 , a 3D convolutional filter is jointly decomposed over a group of spatial atoms and temporal atoms . The two groups of atoms can reconstruct the 3D filter together with the joint coefficients through tensor multiplication . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . A single 3D convolutional layer is now divided into three convolutional layers : a temporal-atom layer to focus on time , a spatial-atom layer for space , and a joint coefficient layer to model the spatiotemporal dependency . Thus , spatial and temporal atoms can now be optimized separately . Different from methods that decorrelate the spatial and temporal modeling , the proposed decomposition can still capture spatiotemporal correlations in the joint coefficients . Moreover , our approach can also significantly reduce model parameters and computation complexity by limiting the number of atoms . One obvious arithmetic property of our joint decomposition is to allow spatial or temporal atoms to be swapped with a set of atoms with the same number but different sizes while keeping the remaining unchanged . For example , we can now achieve tempo-invariance by simply dilating temporal atoms only . To exploit this useful atom-swapping property , we further show how the proposed spatiotemporal jointly decomposed convolutional filter ( STDCF ) permits the direct learning of 3D CNNs with full-size videos . We start with a pedagogical learning decoupling to train a 3D CNN iteratively with full-temporal downsampled-spatial data first , and then with full-spatial downsampled-temporal data . Note that , such simple-minded decoupled learning is less capable of modeling spatiotemporal dependency . A regular 3D CNN model trained from the two-stage strategy above may capture rich temporal features in the first stage , however , such features will be severely degraded by the downsampled-temporal data in the second stage . Exploiting the atom-swapping property of the proposed spatiotemporal joint decomposition , we can instead decouple spatial and temporal learning into iterations of two consecutive sub-stages of learning : temporal-focus stage ( stage-t ) and spatial-focus stage ( stage-s ) . In stage-t , temporal atoms and joint coefficients are learned from full-temporal downsampled-spatial videos by keeping spatial atoms fixed , while in stage-s , spatial atoms and coefficients are updated from full-spatial downsampled-temporal data by keeping temporal atoms fixed . Note that , in stage-s , spatial atoms are learned from full spatial-resolution , while temporal atoms are frozen to prevent degradation from temporal down-sampling , similar for atoms update in stage-t . The proposed two-stage 3D CNN training strategy can model a full temporal and spatial resolution video only from downsampled data while preserving spatiotemporal dependency . Empirically , we show on multiple action recognition datasets , e.g. , KTH [ 19 ] , Kinetics-400 [ 1 ] and Something-Somethingv1 [ 10 ] , that , the STDCF model trained with the proposed strategy significantly reduces memory usage , while producing comparable results with the state-of-the-art models trained with full-size videos . 2 Method . In this section , we start by introducing the proposed spatiotemporal jointly decomposed convolutional filter , STDCF , along with its two properties of capturing spatiotemporal dependency and enabling atom swapping . Then , we decouple spatiotemporal learning into iterative two sub-stage learning . Next , we introduce a simple but effective design , enabled by the atom swapping property , to allow the model to learn from videos with different spatial and temporal resolutions in different sub-stages . 2.1 Spatiotemporal Joint Filter Decomposition . A regular spatiotemporal convolutional filter in deep video models consists of a group of 3D tensors with shape t × k × k for capturing local joint dependency on space and time , where t and k are temporal and spatial filter sizes . The 3D structure not only leads to a dramatic increase in parameters compared to 2D CNNs , but also couples the spatial and temporal learning as each tensor attends to both dimensions simultaneously . To decouple spatial and temporal learning , as well as to reduce model parameters , we propose to jointly decompose a 3D filter over spatial and temporal atoms , as shown in Figure 1 . Our idea originates from spatial filter decomposition as proposed in [ 17 ] , and we extend it to 3D filters by taking the time dimension into consideration . As proposed in [ 17 ] , a 2D spatial filter W ∈ RCout×Cin×k×k ( Cout , Cin are input , output channels , and k is spatial kernel size ) , can be decomposed as a linear combination of M spatial atoms { ψi } Mi=1 , ψi ∈ Rk×k , with coefficients α ∈ RM×Cout×Cin . The spatial convolution then becomes two : A spatial-atom convolution with ψi , a 1 × 1 convolution with joint coefficients α . This decomposition not only reduces the parameters and computation cost but also imposes low-rank filter structures . Then , for a 3D spatiotemporal filter W ∈ RCout×Cin×t×k×k ( t is the temporal kernel size ) , as shown in Figure 1 , we decompose it over spatial atoms Ψ = { ψi } Mi=1 and temporal atoms Φ = { φj } Nj=1 , φj ∈ Rt , with joint coefficients α ∈ RCout×Cin×M×N as W = ∑M i=1 ∑N j=1 α i , jψiφj . In this way , the 3D convolution becomes three , Spatial-atom convolution : Zi ( λ′ , t ) = ∑ u′ I ( λ′ , t , u+ u′ ) ψi ( u ′ ) , Temporal-atom convolution : Yij ( λ′ ) = ∑ t′ φj ( t ′ ) Zi ( λ ′ , t+ t′ ) , 1× 1 joint coefficient convolution : J ( λ , t , u ) = ∑ λ′ ∑ i , j αi , j ( λ′ , λ ) Yij ( λ ′ ) , ( 1 ) where I , J denotes the input and output , u ∈ R2 , t ∈ R , λ , λ′ are spatial and temporal indices , λ , λ′ are output , input channel indices , and Zi , Yij are intermediate outputs of spatial-atom layer and temporal-atom layer . 2.1.1 Spatiotemporal Dependency . Note that a popular alternative way of 3D filter decomposition is to apply rank-1 decomposition to obtain a 2D spatial filter and a 1D temporal filter , as proposed in [ 18 , 26 , 37 ] . Specifically , a 3D filter W ∈ RCin×Cout×t×k×k is decomposed intoWs ∈ RCin×Cout×1×k×k andWt ∈ RCout×Cout×t×1×1 as W = Ws ×Wk . In this way , 3D convolution becomes two : spatial sub-convolution with Ws , and temporal sub-convolution with Wt . However , such a rank-1 decomposition neglects joint spatiotemporal dependency . The spatial subconvolution only focuses on spatial modeling , while the temporal sub-convolution merely attends to temporal modeling . The spatial and temporal features are extracted independently by omitting spatiotemporal correlation . Instead , in our STDCF , joint coefficients α encode spatiotemporal dependency while keeping learning in both dimensions independent . We derive below that STDCF has more capacity than rank-1 decomposition . Recall our joint decomposition of 3D filter W , W ( λ , λ′ ) = ∑ i , j αi , j ( λ , λ′ ) ψi ( u ) φj ( t ) , α i , j ( λ , λ′ ) ∈ R , ( 2 ) λ ∈ [ C ] , λ′ ∈ [ C ′ ] , i ∈ [ M ] , j ∈ [ N ] , and αi , j ( λ , λ′ ) are freely trainable . For each fixed pair of ( λ , λ ′ ) , { αi , j ( λ , λ′ ) } i , j is a M -by-N matrix , and generally is full rank . In the rank-1 decomposition , spatial filters are Ws = W ( λ ′′ , λ′ ) ( u ) , temporal filters are Wt = W ( λ , λ ′′ ) ( t ) , λ′′ ∈ [ C ′′ ] . We can write W ( λ ′′ , λ′ ) s and W ( λ , λ′′ ) t as combination of atoms ψi and φj respectively , W ( λ ′′ , λ′ ) s = ∑ i µi ( λ′′ , λ′ ) ψi , W ( λ , λ′′ ) t = ∑ j νj ( λ , λ′′ ) φj . Tracking the degree of freedom reveals that rank-1 decomposition is more restrictive than our STDCF . Let C = C ′ = C ′′ , α in STDCF has MNC2 many variables if all free . In comparison , µ has MC2 many variables , and ν has NC2 many , thus the rank-1 decomposed convolution formulation has only ( M +N ) C2 many free variables in total . This quantifies the loss of expressiveness from STDCF to rank-1 decomposition . A more comprehensive analysis is provided in Appendix A . To empirically illustrate STDCF is more expressive than rank-1 decomposition , we design a toy example on the translate-rotate-MNIST ( TR-MNIST ) dataset . It consists of short clips of moving digits , which contain complex spatiotemporal correlations . Adopting a 3-layer 3D CNN , we conduct video reconstruction experiments and compare the qualitative and quantitative results between STDCF and rank-1 decomposition . Details of experimental settings are provided in Appendix B . As shown in Figure 2 , given a sequence of moving-rotating digits , STDCF reconstructs in better quality than rank-1 . As for qualitative results , we measure mean PSNR on the test set . STDCF achieves a mean PSNR of 24.60 , whereas the result of rank-1 decomposition is 21.50 , indicating STDCF possesses more capacity to capture spatiotemporal correlations .

Summary: This paper presents a spatiotemporal joint filter decomposition for 3D CNN. Compared with the common 2D+1D decomposition, which is referred to as rank-1 decomposition in the paper, the proposed decomposition has an additional joint coefficients module to encode the spatial and temporal dependency. The proposed STDCF has a larger capacity in theory than the rank-1 decomposition, and is shown to be more friendly to iterative spatial-temporal learning. Toy examples and evaluations on KTH and Kinetics dataset demonstrate the efficiency of STDCF.