V4D: 4D Convolutional Neural Networks for Video-level Representation Learning

1 INTRODUCTION . 3D convolutional neural networks ( 3D CNNs ) and their variants ( Ji et al. , 2010 ; Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Qiu et al. , 2017 ; Wang et al. , 2018b ) provide a simple extension from 2D counterparts for video representation learning . However , due to practical issues such as memory consumption and computational cost , these models are mainly used for clip-level feature learning instead of learning from the whole video . The clip-based methods randomly sample a short clip ( e.g. , 32 frames ) from a video for representation learning , and calculate prediction scores for each clip independently . The prediction scores of all clips are simply averaged to yield the video-level prediction . These clip-based models often ignore the video-level structure and long-range spatiotemporal dependency during training , as they only sample a small portion of the entire video . In fact , in some cases , it could be difficult to identify an action correctly by only using partial observation . Meanwhile , simply averaging the prediction scores of all clips could be sub-optimal during inference . To overcome this issue , Temporal Segment Network ( TSN ) ( Wang et al. , 2016 ) was proposed . TSN uniformly samples multiple clips from the entire video , and the average scores are used to guide back-propagation during training . Thus TSN is a video-level representation learning framework . However , the inter-clip interaction and video-level fusion in TSN is only performed at very late stage , which fails to capture finer temporal structures . In this paper , we propose a general and flexible framework for video-level representation learning , called V4D . As shown in Figure 1 , to model long-range dependency in a more efficient way , V4D is composed of two critical designs : ( 1 ) holistic sampling strategy , and ( 2 ) 4D convolutional interaction . We first introduce a video-level sampling strategy by uniformly sampling a sequence of short-term units covering the whole video . Then we model long-range spatio-temporal dependency by designing a unique 4D residual block . Specifically , we present a 4D convolutional operation to capture inter-clip ∗The corresponding author . interaction , which could enhance the representation power of the original clip-level 3D CNNs . The 4D residual blocks could be easily integrated into the existing 3D CNNs to perform long-range modeling hierarchically , which is more efficient than TSN . We also design a specific video-level inference algorithm for V4D . Finally , we verify the effectiveness of V4D on three video action recognition benchmarks , Mini-Kinetics ( Xie et al. , 2018 ) , Kinetics-400 ( Carreira & Zisserman , 2017 ) and Something-Something-V1 ( Goyal et al. , 2017 ) . Our V4D achieves very competitive performance on the three benchmarks , and obtains evident performance improvement over its 3D counterparts . 2 RELATED WORKS . Two-stream CNNs . Two-stream architecture was originally proposed by ( Simonyan & Zisserman , 2014 ) , where one stream is used for learning from RGB images , and the other one is applied to model optical flow . The results produced by the two streams are then fused at later stages , yielding the final prediction . Two-stream CNNs have achieved impressive results on various video recognition tasks . However , the main limitation is that the computation of optical flow is highly expensive where parallel optimization is difficult to implment , with significant resource explored . Recent effort has been devoted to reducing the computational cost on modeling optical flow , such as ( Dosovitskiy et al. , 2015 ; Sun et al. , 2018 ; Piergiovanni & Ryoo , 2018 ; Zhang et al. , 2016 ) . The two-stream design is a general framework to boost the performance of various CNN models , which is orthogonal to the proposed V4D . 3D CNNs . Recently , 3D CNNs have been proposed ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Wang et al. , 2018a ; b ; Feichtenhofer et al. , 2018 ) . By considering a video as a stack of frames , it is natural to develop 3D convolutions applied directly on video sequence . However , 3D CNNs often introduce a large number of model parameters , which inevitably require a large amount of training data to achieve good performance . As reported in ( Wang et al. , 2018b ; Feichtenhofer et al. , 2018 ) , recent experimental results on large-scale benchmark , likes Kinetics-400 ( Carreira & Zisserman , 2017 ) , show that 3D CNNs can surpass their 2D counterparts in many cases , and even can be on par with or better than the two-stream 2D CNNs . It is noteworthy that most of 3D CNNs are clip-based methods , which only explore a certain part of the holistic video . Long-term Modeling Framework . Various long-term modeling frameworks have been developed for capturing more complex temporal structure for video-level representation learning . In ( Laptev et al. , 2008 ) , video compositional models were proposed to jointly model local video events , where temporal pyramid matching was introduced with a bag-of-visual-words framework to compute longterm temporal structure . However , the rigid composition only works under defined constraints , e.g. , prefixed duration and anchor points provided in time . A mainstream method is to process a continuous video sequence with recurrent neural networks Ng et al . ( 2015 ) ; Donahue et al . ( 2015 ) , where 2D CNNs are used for frame-level feature extraction . Temporal Segment Network ( TSN ) ( Wang et al. , 2016 ) has been proposed to model video-level temporal information with a sparse sampling and aggregation strategy . TSN sparsely samples a set of frames from the whole video , and then the sampled frames are modelled by the same CNN backbone , which outputs a confident score for each frame . The output scores are averaged to generate final video-level prediction . TSN was originally designed for 2D CNNs , but it can be applied to 3D CNNs , which serves as one of the baselines in this paper . One of the main limitations of TSN is that it is difficult to model finer temporal structure due to the average aggregation . Temporal Relational Reasoning Network ( TRN ) ( Zhou et al. , 2018 ) was introduced to model temporal segment relation by encoding individual representation of each segment with relation networks . TRN is able to model video-level temporal order but lacks the capacity of capturing finer temporal structure . The proposed V4D can outperform these previous video-level learning methods on both appearance-dominated video recognition ( e.g. , on Kinetics ) and motion-dominated video recognition ( e.g. , on Something-Something ) . It is able to model both short-term and long-term temporal structure with a unique design of 4D residual blocks . 3 VIDEO-LEVEL 4D COVOLUTIONAL NEURAL NETWORKS . In this section , we introduce new Video-level 4D Convolution Neural Networks , namely V4D , for video action recognition . This is the first attempt to design 4D convolutions for RGB-based video recognition . Previous methods , such as You & Jiang ( 2018 ) ; Choy et al . ( 2019 ) , utilize 4D CNNs to process videos of point cloud by using 4D data as input . Instead , our V4D processes videos of RGB frames with input of 3D data . Existing 3D CNNs often take a short-term snippet as input , without considering the evolution of 3D spatio-temporal features for video-level representation . In Wang et al . ( 2018b ) ; Yue et al . ( 2018 ) ; Liu et al . ( 2019 ) , self-attention mechanism was developed to model non-local spatio-temporal features , but these methods were originally designed for clip-based 3D CNNs . It remains unclear how to incorporate such operations on holistic video representation , and whether such operations are useful for video-level representation learning . Our goal is to model 3D spatio-temporal features globally , which can be implemented in a higher dimension . In this work , we introduce new Residual 4D Blocks , which allow us to cast 3D CNNs into 4D CNNs for learning long-range interactions of the 3D features , resulting in a “ time of time ” video-level representation . 3.1 A VIDEO-LEVEL SAMPLING STRATEGY . To model meaningful video-level representation for action recognition , the input to the networks has to cover the holistic duration of a given video , and at the same time preserve short-term action details . A straightforward approach is to implement per-frame training of the networks yet this is not practical by considering the limit of computation resource . In this work , we uniformly divide the whole video into U sections , and select a snippet from each section to represent a short-term action pattern , called “ action unit ” . Then we have U action units to represent the holistic action in a video . Formally , we denote the video-level input V = { A1 , A2 , ... , AU } , where Ai ∈ RC×T×H×W . During training , each action unit Ai is randomly selected from each of the U sections . During testing , the center of each Ai locates exactly at the center of the corresponding section . 3.2 4D CONVOLUTIONS FOR LEARNING SPATIO-TEMPORAL INTERACTIONS . 3D Convolutional kernels have been proposed , and are powerful to model short-term spatio-temporal features . However , the receptive fields of 3D kernels are often limited due to the small sizes of kernels , and pooling operations are applied to enlarge the receptive fields , resulting in a significant cost of information loss . This inspired us to develop new operations which are able to model both short- and long-term spatio-temporal representations simultaneously , with easy implementations and fast training . From this prospective , we propose 4D convolutions for better modeling the long-range spatio-temporal interactions . Specifically , we denote the input to 4D convolutions as a tensor V of size ( C , U , T , H , W ) , where C is number of channel , U is the number of action units ( the 4-th dimension in this paper ) , T , H , W are temporal length , height and width of an action unit . We omit the batch dimension for simplicity . By following the annotations provided in Ji et al . ( 2010 ) , a pixel at position ( u , t , h , w ) of the jth channel in the output is denoted as outhwj , and a 4D convolution operation can be formulated as : outhwj = bj + Cin∑ c S−1∑ s=0 P−1∑ p=0 Q−1∑ q=0 R−1∑ r=0 Wspqrjc v ( u+s ) ( t+p ) ( h+q ) ( w+r ) c ( 1 ) where bj is a bias term , c is one of the Cin input channels of the feature maps from input V , S×P ×Q×R is the shape of 4D convolutional kernel , Wspqrjc is the weight at the position ( s , p , q , r ) of the kernel , corresponding to the c-th channel of the input feature maps and j-th channel of the output feature maps . Convolutional operation is linear , and the sequential sum operations in E.q . 1 are exchangeable . Thus we can generate E.q . 2 , where the expression in the parentheses can be implemented by 3D convolutions , allowing us to implement 4D convolutions using 3D convolutions , while most deep learning libraries do not directly provide 4D convolutional operations . outhwj = bj + S−1∑ s=0 ( Cin∑ c P−1∑ p=0 Q−1∑ q=0 R−1∑ r=0 Wspqrjc v ( u+s ) ( t+p ) ( h+q ) ( w+r ) c ) ( 2 ) With the 4D convolutional kernel , the short-term 3D features of an individual action unit and longterm temporal evolution of multiple action units can be modeled simultaneously in the 4D space . Compared to 3D convolutions , the proposed 4D convolutions are able to model videos in a more meaningful 4D feature space that enables it to learn more complicated interactions of long-range 3D spatio-temporal representation . However , 4D convolutions inevitably introduce more parameters and computation cost . For example , a 4D convolutional kernel of k × k × k × k employs k times more parameters than a 3D kernel of k × k × k. In practice , we explore k × k × 1× 1 and k × 1× 1× 1 kernels , to reduce the parameters and avoid the risk of overfitting . The implementation of different kernels is shown in Figure 2 .

Summary: The paper presents a video classification framework that employs 4D convolution to capture longer term temporal structure than the popular 3D convolution schemes. This is achieved by treating the compositional space of local 3D video snippets as an individual dimension where an individual convolution is applied. The 4D convolution is integrated in resnet blocks and implemented via first applying 3D convolution to regular spatio-temporal video volumes and then the compositional space convolution, to leverage existing 3D operators. Empirical evaluation on three benchmarks against other baselines suggested the advantage of the proposed method.