VA-RED$^2$: Video Adaptive Redundancy Reduction

1 INTRODUCTION . Large computationally expensive models based on 2D/3D convolutional neural networks ( CNNs ) are widely used in video understanding ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Tran et al. , 2018 ) . Thus , increasing computational efficiency is highly sought after ( Feichtenhofer , 2020 ; Zhou et al. , 2018c ; Zolfaghari et al. , 2018 ) . However , most of these efficient approaches focus on architectural changes in order to maximize network capacity while maintaining a compact model ( Zolfaghari et al. , 2018 ; Feichtenhofer , 2020 ) or improving the way that the network consumes temporal information ( Feichtenhofer et al. , 2018 ; Korbar et al. , 2019 ) . Despite promising results , it is well known that CNNs perform unnecessary computations at some levels of the network ( Han et al. , 2015a ; Howard et al. , 2017 ; Sandler et al. , 2018 ; Feichtenhofer , 2020 ; Pan et al. , 2018 ) , especially for video models since the high appearance similarity between consecutive frames results in a large amount of redundancy . In this paper , we aim at dynamically reducing the internal computations of popular video CNN architectures . Our motivation comes from the existence of highly similar feature maps across both time and channel dimensions in video models . Furthermore , this internal redundancy varies depending on the input : for instance , static videos will have more temporal redundancy whereas videos depicting a single large object moving tend to produce a higher number of redundant feature maps . To reduce the varied redundancy across channel and temporal dimensions , we introduce an input-dependent redundancy reduction framework called VA-RED2 ( Video Adaptive REDundancy REDuction ) for efficient video recognition ( see Figure 1 for an illustrative example ) . Our method is model-agnostic and hence can be applied to any state-of-the-art video recognition networks . The key mechanism that VA-RED2 uses to increase efficiency is to replace full computations of some redundant feature maps with cheap reconstruction operations . Specifically , our framework avoids computing all the feature maps . Instead , we choose to only calculate those non-redundant part of feature maps and reconstruct the rest using cheap linear operations from the non-redundant features maps . In addition , VA-RED2 makes decisions on a per-input basis : our framework learns an input-dependent policy that defines a ” full computation ratio ” for each layer of a 2D/3D network . This ratio determines the amount of features that will be fully computed at that layer , versus the features that will be reconstructed from the non-redundant feature maps . Importantly , we apply this strategy on both time and channel dimensions . We show that for both traditional video models such as I3D ( Carreira & Zisserman , 2017 ) , R ( 2+1 ) D ( Tran et al. , 2018 ) , and more advanced models such as X3D ( Feichtenhofer , 2020 ) , this method significantly reduces the total floating point operations ( FLOPs ) on common video datasets without accuracy degradation . The main contributions of our work includes : ( 1 ) A novel input-dependent adaptive framework for efficient video recognition , VA-RED2 , that automatically decides what feature maps to compute per input instance . Our approach is in contrast to most current video processing networks , where feature redundancy across both time and channel dimensions is not directly mitigated . ( 2 ) An adaptive policy jointly learned with the network weights in a fully differentiable way with a sharedweight mechanism , that allows us to make decisions on how many feature maps to compute . Our approach is model-agnostic and can be applied to any backbones to reduce feature redundancy in both time and channel domains . ( 3 ) Striking results of VA-RED2 over baselines , with a 30 % reduction in computation in comparison to R ( 2+1 ) D ( Tran et al. , 2018 ) , a 40 % over I3D-InceptionV2 ( Carreira & Zisserman , 2017 ) , and about 20 % over the recently proposed X3D-M ( Feichtenhofer , 2020 ) without any performance loss , for video action recognition task . The superiority of our approach is extensively tested on three video recognition datasets ( Mini-Kinetics-200 , Kinetics-400 ( Carreira & Zisserman , 2017 ) , and Moments-In-Time ( Monfort et al. , 2019 ) ) and one spatio-temporal action localization dataset ( J-HMDB-21 ( Jhuang et al. , 2013 ) ) . ( 4 ) A generalization of our framework to video action recognition , spatio-temporal localization , and semantic segmentation tasks , achieving promising results while offering significant reduction in computation over competing methods . 2 RELATED WORK . Efficiency in Video Understanding Models . Video understanding has made significant progress in recent years , mainly due to the adoption of convolutional neural networks , in form of 2D CNNs ( Karpathy et al. , 2014 ; Simonyan & Zisserman , 2014 ; Chéron et al. , 2015 ; Feichtenhofer et al. , 2017 ; Gkioxari & Malik , 2015 ; Wang et al. , 2016 ; Zhou et al. , 2018a ; Lin et al. , 2019 ; Fan et al. , 2019 ) or 3D CNNs ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Hara et al. , 2018 ; Tran et al. , 2018 ) . Despite promising results on common benchmarks , there is a significant interest in developing more efficient techniques and smaller models with reasonable performance . Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures ( Xie et al. , 2018 ; Zhou et al. , 2018c ; Zolfaghari et al. , 2018 ) , group convolution ( Tran et al. , 2019 ) or selecting salient clips ( Korbar et al. , 2019 ) . Feichtenhofer et al. , ( Feichtenhofer et al. , 2018 ) propose a dedicated low-framerate pathway . Expansion of 2D architectures through a stepwise expansion approach over the key variables such as temporal duration , frame rate , spatial resolution , network width , is recently proposed in ( Feichtenhofer , 2020 ) . Diba et al . ( Diba et al. , 2019 ) learn motion dynamic of videos with a self-supervised task for video understanding . Fan et al . ( Fan et al. , 2020 ) incorporate a efficient learnable 3D-shift module into a 3D video network . Wang et al . ( Wang et al. , 2020 ) devise a correlation module to learn correlation along temporal dimension . Li et al . ( Li et al. , 2020 ) encode the clip-level ordered temporal information with a CIDC network . While these approaches bring considerable efficiency improvements , none of them dynamically calibrates the required feature map computations on a per-input basis . Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input . Adaptive Inference . Many adaptive computation methods have been recently proposed with the goal of improving efficiency ( Bengio et al. , 2015 ; 2013 ; Veit & Belongie , 2018 ; Wang et al. , 2018 ; Graves , 2016 ; Meng et al. , 2021 ) . Several works have been proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference ( Yu et al. , 2018 ; Figurnov et al. , 2017 ; McGill & Perona , 2017 ; Teerapittayanon et al. , 2016 ) Wang et al . ( Wang et al. , 2018 ) propose to skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training . Veit et al . ( Veit & Belongie , 2018 ) propose a block skipping method controlled by samples from a Gumbel softmax , while Wu et al . ( Wu et al. , 2018 ) develop a reinforcement learning approach to achieve this goal . Adaptive computation time for recurrent neural networks is also presented in ( Graves , 2016 ) . SpotTune ( Guo et al. , 2019 ) learns to adaptively route information through finetuned or pre-trained layers . A few works have been recently proposed for selecting salient frames conditioned on the input ( Yeung et al. , 2016 ; Wu et al. , 2019 ; Korbar et al. , 2019 ; Gao et al. , 2019 ) while recognizing actions in long untrimmed videos . Different from adaptive data sampling ( Yeung et al. , 2016 ; Wu et al. , 2019 ; Korbar et al. , 2019 ; Gao et al. , 2019 ) , in this paper , our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis , for efficient video recognition . AR-Net ( Meng et al. , 2020 ) recently learns to adaptively choose the resolution of input frames with several individual backbone networks for video inference . In contrast , our method focuses on reducing the redundancy in both temporal and channel dimension , and is applicable to both 3D and 2D models , while AR-Net is only for 2D model and it is focused on spatial resolution . Moreover , our method integrates all the inference routes into a single model which is in the almost same size to the original base model . Thus our model is significantly smaller than AR-Net in terms number of model parameters . Neural Architecture Search . Our network learns the best internal redundancy reduction scheme , which is similar to previous work on automatically searching architectures ( Elsken et al. , 2018 ) . Liu et al . ( Liu et al. , 2018 ) formulate the architecture search task in a differentiable manner ; Cai et al . ( Cai et al. , 2018 ) directly learn architectures for a target task and hardware , Tan et al . ( Tan & Le , 2019 ) design a compound scaling strategy that searches through several key dimensions for CNNs ( depth , width , resolution ) . Finally , Tan et al . ( Tan et al. , 2019 ) incorporate latency to find efficient networks adapted for mobile use . In contrast , our approach learns a policy that chooses over full or reduced convolutions at inference time , effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy . 3 VIDEO ADAPTIVE REDUNDANCY REDUCTION . Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation . The intuition behind our proposed method is that there are many similar feature maps along the temporal and channel dimensions . For each video instance , we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension . Then , for the other feature maps , we reconstruct them from those pre-computed feature maps using cheap linear operations . Approach Overview . Without loss of generality , we start from a 3D convolutional network G , and denote its lth 3D convolution layer as fl , and the corresponding input and output feature maps as Xl and Yl respectively . For each 3D convolution layer , we use a very lightweight policy layer pl denoted as soft modulation gate to decide the ratio of feature maps along the temporal and channel dimensions which need to be computed . As shown in Figure 2 , for temporal-wise dynamic inference , we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with a factor R = 2pl ( Xl ) [ 0 ] . Thus the shape of output Y ′l becomes Cout × To/R ×Ho ×Wo . To keep the same output shape , we reconstruct the remaining features based on Y ′l as Yl [ j + iR ] = { Φti , j ( Y ′ l [ i ] ) if j ∈ { 1 , ... , R− 1 } Y ′l [ i ] if j = 0 , i ∈ { 0 , 1 , ... , To/R− 1 } , ( 1 ) where Yl [ j + iR ] represents the ( j + iR ) th feature map of Yl along the temporal dimension , Y ′l [ i ] denotes the ith feature map of Y ′l , and Φ t i , j is the cheap linear operation along the temporal dimension . The total computational cost of this process can be written as : C ( f tl ) = 1 R · C ( fl ) + ∑ i , j C ( Φti , j ) ≈ 1 R · C ( fl ) , ( 2 ) where the function C ( · ) returns the computation cost for a specific operation , and f tl represents our dynamic convolution process along temporal dimension . Different from temporal-wise dynamic inference , we reduce the channel-wise computation by dynamically controlling the number of output channels . We scale the output channel number with a factor r = ( 12 ) pl ( Xl ) [ 1 ] . In this case , the shape of output Y ′l is rCout × To × Ho × Wo . Same as before , we reconstruct the remaining features via cheap linear operations , which can be formulated as Yl = [ Y ′l , Φ c ( Y ′l ) ] , where Φc ( Y ′l ) ∈ R ( 1−r ) Cout×To×Ho×Wo represents the cheaply generated feature maps along the channel dimension , and Yl ∈ RCout×To×Ho×Wo is the output of the channel-wise dynamic inference . The total computation cost of joint temporal-wise and channel-wise dynamic inference is : C ( f t , cl ) ≈ r R · C ( fl ) , ( 3 ) where f t , cl is the adjunct process of temporal-wise and channel-wise dynamic inference . Soft Modulation Gate for Differentiable Optimization . We adopt an extremely lightweight policy layer pl called soft modulation gate for each convolution layer fl to modulate the ratio of features which need to be computed . Specifically , the soft modulation gate takes the input feature maps Xl as input and learns two probability vectors V lt ∈ RSt and V lc ∈ RSc , where St and Sc are the temporal search space size and the channel search space size respectively . The V lt and V l c are learned by : [ V lt , V l c ] = pl ( Xl ) = φ ( F ( ωp,2 , δ ( N ( F ( ωp,1 , G ( Xl ) ) ) ) ) + βlp ) , ( 4 ) where F ( · , · ) denotes the fully-connected layer , N is the batch normalization , δ ( · ) represents the tanh ( · ) function , G is the global pooling operation whose output shape is Cin · T × 1× 1 , φ ( · ) is the output activation function , here we just use max ( tanh ( · ) , 0 ) whose output range is [ 0 , 1 ) , and ωp,1 ∈ R ( St+Sc ) ×Dh , ωp,2 ∈ RDh×Cin·T are the weights of their corresponding layers , Dh is the hidden dimension number . V lt and V l c will then be used to modulate the ratio of the feature maps to be computed in temporal-wise dynamic convolution and channel-wise dynamic convolution . During training , we obtain the final output of the dynamic convolution by weighted sum of all the feature maps which contains different ratio of fully-computed features as follows : Y lc = Sc∑ i=1 V lc [ i ] · f cl ( Xl , r = ( 1 2 ) ( i−1 ) ) , Yl = St∑ j=1 V lt [ j ] · f tl ( Y lc , R = 2 ( j−1 ) ) , ( 5 ) where f cl ( · , r ) is the channel-wise dynamic convolution with the channel scaling factor r , and f tl ( · , R ) it the temporal-wise dynamic convolution with the temporal stride scaling factor R. During the inference phase , only the dynamic convolutions whose weights are not zero will be computed . Shared-weight Training and Inference . Many works in adaptive computation and neural architecture search suffer from very heavy computational cost and memory usage during training stage due to the large search space . In our case , under the naive implementation , the training computational cost and parameter size would linearly grow as the search space size increases . To train our model efficiently , we utilize a weight-sharing mechanism to reduce the computational cost and training memory . To be specific , we first compute all the possible necessary features using a big kernel . Then , for each dynamic convolution with different scaling factor , we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output . Though this , we are able to keep the computational cost at a constant value invariant to the search space . More details on this are included in Section B of the Appendix . Efficiency Loss . To encourage our network to output a computational efficient subgraph , we introduce the efficiency loss Lc during the training process , which can be formulated as Le = ( µ0 L∑ l=1 C ( fl ) ∑L k=1 C ( fk ) · r s l Rsl ) 2 , µ0 = { 1 if correct 0 otherwise , ( 6 ) where rsl is channel scaling factor of the largest filter in the series of channel-wise dynamic convolutions , and Rsl is stride scaling factor of the largest filter of temporal-wise dynamic convolutions . Overall , the loss function of our whole framework can be written as L = La + λeLe , where La is the accuracy loss of the whole network and λe is the weight of efficiency loss which can be used to balance the importance of the optimization of prediction accuracy and computational cost .

Summary: The paper presents a framework to reduce internal redundancy in the video recognition model. To do so, given the input frames, the framework predicts two scaling factors to conduct temporal and channel dimension reduction. The remaining part is reconstructed by cheap operations. The authors show that the framework achieves favorable results on several benchmarks.