Dynamic Normalization and Relay for Video Action Recognition

1 Introduction . Human action recognition is a fundamental problem in video understanding , which has been studied for decades . The performance of an intelligent action recognition system depends on how well it can extract compact and discriminative features to characterize temporal evolutions of human object appearance and its motion information in videos . Early seminal methods [ 29 , 11 , 8 , 46 , 28 , 45 , 58 ] ubiquitously use hand-crafted features to construct spatial-temporal descriptors . In recent years , deep learning based models [ 70 ] have become the mainstream in action recognition research , mainly due to their remarkably better representation and generalization abilities compared to conventional methods , especially to model large amounts of training data . Despite the prevalence of deep learning based methods for action recognition , learning efficient yet effective video representations is still a challenging problem . To improve the spatial-temporal * The first two authors contributed equally to the writing of the paper . † Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . representation learning of CNNs for action recognition , we investigate a new technical perspective by presenting an improved normalization design called Dynamic Normalization and Relay ( DNR ) , learning to predict accurate layer-wise normalization parameters for any action recognition network in an input-dependent manner . We are motivated by two plain facts . On the one side , due to the problem of space-time cube data structure , the huge memory and compute demand in training a popular network for action recognition usually restricts the batch size to a much smaller range compared to the settings for image classification tasks . For instance , on Kinetics [ 5 ] the batch size is typically set to 32 video clips of length 8 , while on ImageNet [ 44 ] it is set to 256 images . On the other side , existing CNN models for video action recognition use Batch Normalization ( BN ) [ 24 ] as a standard component to normalize feature maps learnt at each layer . Although the significance of BN has been well demonstrated in many previous action recognition works , it will easily introduce noise during the estimation of layer-wise normalization parameters when the batch size is small , degenerating the accuracy of the trained model to some extent . This problem could be alleviated by using recent normalization methods [ 23 , 63 , 26 , 57 , 31 , 38 , 67 ] . However , the performance gain of doing so is limited as they are primarily proposed for image recognition tasks . For example , on Kinetics with 32-frame video clips , it only brings -0.3 % and 0.7 % top-1 accuracy improvement to ResNet-50 C3D baseline [ 53 ] trained with a batch size of 8 and 4 video clips per GPU by replacing BN with Group Normalization ( GN ) , as reported in [ 63 ] . We argue this is mainly because a direct extension of them from image to video domain lacks a proper mechanism to handle complicated spatial-temporal feature variations of video data . However , to the best of our knowledge , there is almost no research effort made to explore a better normalization mechanism for promoting the training of existing action recognition networks with video clip inputs . As an improved video normalization design for action recognition networks , our DNR considers . three questions at multiple representation learning scales to alleviate the aforementioned gap : ( 1 ) At individual frame scale , how to estimate input-dependent layer-wise normalization parameters ? ( 2 ) At time scale , how to enhance the dependencies of the estimated normalization parameters for layer-specific feature representations between neighboring frames ? ( 3 ) At network depth scale , how to enhance the clip-level dependencies of the estimated normalization parameters between neighboring layers ? To the first question , we formulate our DNR as a dynamic normalization predication scheme [ 1 ] where normalization parameters for any layer of an action recognition network are learnt and generated dynamically both in training and inference , making them to be input-dependent . To the other two questions , we bridge the DNR formulation with two interdependent normalization relay modules called cross-temporal DNR and cross-layer DNR , which are encapsulated into a light-weight recurrent structure [ 20 ] . For a certain layer , the normalization parameters learnt by cross-temporal DNR module are conditioned on the input features as well as the normalization parameters estimated from the neighboring frames based features at the same layer , while the normalization parameters learnt by cross-layer DNR module are conditioned on the input features as well as the normalization parameters estimated from the whole video clip based features at the preceding layers . On the one side , the dynamic normalization relay along the temporal axis models the layer-specific frame-level correlations of the spatial-temporal feature distributions between neighboring frames . On the other side , the dynamic normalization relay along the sequential layers considers all the hidden layers as a whole system , and estimates the video clip-level feature dynamics of the current layer by jointly considering the feature distributions of its preceding layers . Benefiting from the above complementary designs , our DNR can significantly improve the performance of existing action recognition networks via replacing their normalization layers by two types of DNR modules . Experimental results on Kinetics and Something-Something datasets [ 18 , 40 ] show that by applying DNR to prevailing 2D CNN backbones such as ResNet [ 19 ] , ResNeXt [ 64 ] and BNInception [ 25 ] , it brings large accuracy improvements to the baselines . We also apply DNR to 3D CNN backbones and several latest 2D CNN designs constructed with sophisticated spatial-temporal modules for further validation of its effectiveness . Thorough ablative experiments are also conducted to have a deep analysis of the proposed method . 2 Related Works . In this section , we summarize existing works that are related to our method , and discuss their connections and differences . Efficient Spatial-Temporal Modeling . Efficient spatial-temporal modeling is a hot research topic in video action recognition . To strike a direct trade-off between network accuracy and efficiency , some works [ 42 , 54 , 37 ] use factorized convolutions to approximate 3D convolutions , and some works [ 65 , 71 , 15 ] mix 2D and 3D convolutions into a single CNN architecture . Two-path networks [ 16 , 14 , 43 , 30 ] are another kind of efficient 3D designs . One representative is SlowFast [ 16 ] which uses a slow pathway operating at low frame rate of high image resolution to capture spatial semantics and a fast pathway operating at high frame rate of low image resolution to capture motion information . Compared to 2D CNN counterparts , the aforementioned networks mixing 2D and 3D convolutions as well as two-path networks with 3D convolutions are not runtime efficient . They are efficient only to their own motivations . In some recent works [ 34 , 51 , 47 , 13 , 39 , 59 ] , switchable channel grouping operations such as shift and shuffle along the temporal dimension are incorporated into either 2D or 3D CNN networks to promote the information exchange between neighboring frames . As our method improves the basic normalization strategy for training action recognition networks , it could be potentially combined with some of these efficient designs to get improved performance . Action Recognition with Recurrent Models . Thanks to their natural abilities to model temporal relations within video data , Recurrent Neural Networks ( RNNs ) instantiated as Long Short Term Memory ( LSTM ) networks [ 20 ] are popularly used in action recognition . This line of research is originally explored by [ 3 ] in which an LSTM network is used to classify actions in soccer videos , taking hand-crafted features as the input . In the deep learning era , a lot of architectural extensions [ 69 , 12 , 50 , 17 , 48 , 52 , 41 , 4 , 55 , 33 ] have been presented , using LSTM networks to aggregate the frame-level CNN features for video clip-level action predications . These methods follow a basic CNN-LSTM framework , using LSTM networks as the fusion module to model temporal dynamics over the frame-level features extracted by one or several pre-trained CNN models . Unlike them , we investigate a new perspective for the usage of LSTM : we bridge a light-weight LSTM structure with a dynamic normalization formulation , learning to generate frame-adaptive layer-wise normalization parameters for improving the performance of existing action recognition networks . Action Recognition with Attentive Models . There are a lot of existing works which use different attention blocks to boost the spatial-temporal representation learning of CNNs for action recognition . [ 61 ] uses a self-attention block with non-local operations to capture long-range dependencies . Several recent methods [ 35 , 32 , 9 ] extend the attention designs of the SENet family [ 21 , 62 ] to construct attention blocks for video action recognition , among which [ 35 ] addresses the attention block design for 2D CNNs while [ 32 ] and [ 9 ] are for 3D CNNs . Besides these attention blocks , dynamic convolutions [ 27 , 68 , 6 ] are used in [ 10 , 36 ] to build their attention blocks . [ 36 ] presents a temporal adaptive module to generate video-specific temporal kernels . Compared to these action recognition methods inspired by SENet and dynamic convolutions , our method shares the similarity of modulating learnt features dynamically to improve the spatial-temporal representation learning . However , they focus on designing powerful attention blocks to replace existing blocks of an action recognition network , while our method introduces two drop-in normalization relay modules built with a light-weight recurrent structure in a dynamic video normalization perspective . Normalization Methods . For modern CNNs , Batch Normalization ( BN ) [ 24 ] is a standard component , improving the optimization stability and accelerating the training convergence . Despite its great success , BN is prone to making an inaccurate estimation of the batched statistics when training with a small batch size , exhibiting a noticeable accuracy drop . To alleviate this issue , a lot of variant methods have been proposed . Batch Renormalization ( BRN ) [ 23 ] introduces two extra hyperparameters to limit the estimated batch statistics . Eval Normalization ( EN ) [ 49 ] presents an online strategy to estimate corrected statistics for BN . Moving Average Batch Normalization ( MABN ) [ 67 ] uses moving average statistics in both forward and backward propagation . These three methods concentrate on improving estimated statistics , particularly mean and standard deviation for each channel . Recurrent Batch Normalization [ 7 ] utilizes the benefits of BN to normalize the hidden states of LSTM networks . Switchable Normalization ( SN ) [ 57 ] learns to automatically select the best choice for each layer among a predefined set of normalizers including BN , Layer Normalization ( LN ) [ 2 ] and Instance Normalization ( IN ) [ 56 ] . Group Normalization ( GN ) [ 63 ] divides the channels into groups and computes the group-wise batched statistics . Adaptive Normalization ( AdaNorm ) [ 66 ] controls the scaling weight of LN with a simple linear transformation . Batch Kalman Normalization ( BKN ) [ 57 ] considers layer correlations when computing statistics of a certain BN layer . Following the basic principle of BKN to model layer correlations , DIANet [ 22 ] uses an LSTM network instead of Kalman Filtering process to improve the design of convolutional attention blocks for image classification tasks . However , DIANet leads to noticeable top-1 accuracy drops ( over 3 % even on CIFAR-100 dataset ) when using its attention blocks to replace batch normalization layers . Recently , there are also some works attempting to make the normalization be sample-adaptive . Instance-Level Meta Normalization ( ILM ) [ 26 ] learns to predict the normalization parameters adaptively with an encoderdecoder sub-network . Attentive Normalization [ 31 ] makes an integration of feature normalization and channel-wise feature attention [ 21 ] . As discussed in the previous section , these normalization methods are mainly proposed for image recognition tasks , while this work explores the dynamic normalization mechanism to promote the training of action recognition networks at multiple scales .

Summary: The method proposes a module to dynamically scale the tensors within a temporal action recognition model that can act as a drop-in addition to existing models. To achieve this, the per-frame input tensors are pooled into vectors, and a LSTM is used to condition the linear scaling of the tensor depending on the (avgpooled) representation of the current frame, but also previous predictions. This is also extended to include the predictions from the previous layer. There are also a couple of tweaks to the LSTM to improve performance.