AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition

1 INTRODUCTION . Over the last few years , video action recognition has made rapid progress with the introduction of a number of large-scale video datasets ( Carreira & Zisserman , 2017 ; Monfort et al. , 2018 ; Goyal et al. , 2017 ) . Despite impressive results on commonly used benchmark datasets , efficiency remains a great challenge for many resource constrained applications due to the heavy computational burden of deep Convolutional Neural Network ( CNN ) models . Motivated by the need of efficiency , extensive studies have been recently conducted that focus on either designing new lightweight architectures ( e.g. , R ( 2+1 ) D ( Tran et al. , 2018 ) , S3D ( Xie et al. , 2018 ) , channel-separated CNNs ( Tran et al. , 2019 ) ) or selecting salient frames/clips conditioned on the input ( Yeung et al. , 2016 ; Wu et al. , 2019b ; Korbar et al. , 2019 ; Gao et al. , 2020 ) . However , most of the existing approaches do not consider the fact that there exists redundancy in CNN features which can significantly save computation leading to more efficient action recognition . In particular , orthogonal to the design of compact models , the computational cost of a CNN model also has much to do with the redundancy of CNN features ( Han et al. , 2019 ) . Furthermore , the amount of redundancy depends on the dynamics and type of events in the video : A set of still frames for a simple action ( e.g . “ Sleeping ” ) will have a higher redundancy comparing to a fast-changed action with rich interaction and deformation ( e.g . “ Pulling two ends of something so that it gets stretched ” ) . Thus , based on the input we could compute just a subset of features , while the rest of the channels can reuse history feature maps or even be skipped without losing any accuracy , resulting in large computational savings compared to computing all the features at a given CNN layer . Based on this intuition , we present a new perspective for efficient action recognition by adaptively deciding what channels to compute or reuse , on a per instance basis , for recognizing complex actions . In this paper , we propose AdaFuse , an adaptive temporal fusion network that learns a decision policy to dynamically fuse channels from current and history feature maps for efficient action recognition . Specifically , our approach reuses history features when necessary ( i.e. , dynamically decides which channels to keep , reuse or skip per layer and per instance ) with the goal of improving both recognition ∗Email : mengyuethu @ gmail.com . This work was done while Yue was an AI Resident at IBM Research . accuracy and efficiency . As these decisions are discrete and non-differentiable , we rely on a Gumbel Softmax sampling approach ( Jang et al. , 2016 ) to learn the policy jointly with the network parameters through standard back-propagation , without resorting to complex reinforcement learning as in ( Wu et al. , 2019b ; Fan et al. , 2018 ; Yeung et al. , 2016 ) . We design the loss to achieve both competitive performance and resource efficiency required for action recognition . Extensive experiments on multiple benchmarks show that AdaFuse significantly reduces the computation without accuracy loss . The main contributions of our work are as follows : • We propose a novel approach that automatically determines which channels to keep , reuse or skip per layer and per target instance for efficient action recognition . • Our approach is model-agnostic , which allows this to be served as a plugin operation for a wide range of 2D CNN-based action recognition architectures . • The overall policy distribution can be seen as an indicator for the dataset characteristic , and the block-level distribution can bring potential guidance for future architecture designs . • We conduct extensive experiments on four benchmark datasets ( Something-Something V1 ( Goyal et al. , 2017 ) , Something-Something V2 ( Mahdisoltani et al. , 2018 ) , Jester ( Materzynska et al. , 2019 ) and Mini-Kinetics ( Kay et al. , 2017 ) ) to demonstrate the superiority of our proposed approach over state-of-the-art methods . 2 RELATED WORK . Action Recognition . Much progress has been made in developing a variety of ways to recognize complex actions , by either applying 2D-CNNs ( Karpathy et al. , 2014 ; Wang et al. , 2016 ; Fan et al. , 2019 ) or 3D-CNNs ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Hara et al. , 2018 ) . Most successful architectures are usually based on the two-stream model ( Simonyan & Zisserman , 2014 ) , processing RGB frames and optical-flow in two separate CNNs with a late fusion in the upper layers ( Karpathy et al. , 2014 ) or further combining with other modalities ( Asghari-Esfeden et al. , 2020 ; Li et al. , 2020a ) . Another popular approach for CNN-based action recognition is the use of 2D-CNN to extract frame-level features and then model the temporal causality using different aggregation modules such as temporal averaging in TSN ( Wang et al. , 2016 ) , a bag of features scheme in TRN ( Zhou et al. , 2018 ) , channel shifting in TSM ( Lin et al. , 2019 ) , depthwise convolutions in TAM ( Fan et al. , 2019 ) , non-local neural networks ( Wang et al. , 2018a ) , temporal enhancement and interaction module in TEINet ( Liu et al. , 2020 ) , and LSTMs ( Donahue et al. , 2015 ) . Many variants of 3D-CNNs such as C3D ( Tran et al. , 2015 ; Ji et al. , 2013 ) , I3D ( Carreira & Zisserman , 2017 ) and ResNet3D ( Hara et al. , 2018 ) , that use 3D convolutions to model space and time jointly , have also been introduced for action recognition . SlowFast ( Feichtenhofer et al. , 2018 ) employs two pathways to capture temporal information by processing a video at both slow and fast frame rates . Recently , STM ( Jiang et al. , 2019 ) proposes new channel-wise convolutional blocks to jointly capture spatio-temporal and motion information in consecutive frames . TEA ( Li et al. , 2020b ) introduces a motion excitation module including multiple temporal aggregation modules to capture both short- and long-range temporal evolution in videos . Gate-Shift networks ( Sudhakaran et al. , 2020 ) use spatial gating for spatial-temporal decomposition of 3D kernels in Inception-based architectures . While extensive studies have been conducted in the last few years , limited efforts have been made towards efficient action recognition ( Wu et al. , 2019b ; a ; Gao et al. , 2020 ) . Specifically , methods for efficient recognition focus on either designing new lightweight architectures that aim to reduce the complexity by decomposing the 3D convolution into 2D spatial convolution and 1D temporal convolution ( e.g. , R ( 2+1 ) D ( Tran et al. , 2018 ) , S3D ( Xie et al. , 2018 ) , channel-separated CNNs ( Tran et al. , 2019 ) ) or selecting salient frames/clips conditioned on the input ( Yeung et al. , 2016 ; Wu et al. , 2019b ; Korbar et al. , 2019 ; Gao et al. , 2020 ) . Our approach is most related to the latter which focuses on conditional computation and is agnostic to the network architecture used for recognizing actions . However , instead of focusing on data sampling , our approach dynamically fuses channels from current and history feature maps to reduce the computation . Furthermore , as feature maps can be redundant or noisy , we use a skipping operation to make it more efficient for action recognition . Conditional Computation . Many conditional computation methods have been recently proposed with the goal of improving computational efficiency ( Bengio et al. , 2015 ; 2013 ; Veit & Belongie , 2018 ; Wang et al. , 2018b ; Graves , 2016 ; Meng et al. , 2020 ; Pan et al. , 2021 ) . Several works have been proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference ( Figurnov et al. , 2017 ; McGill & Perona , 2017 ; Wu et al. , 2020 ) . BlockDrop ( Wu et al. , 2018 ) effectively reduces the inference time by learning to dynamically select which layers to execute per sample during inference . SpotTune ( Guo et al. , 2019 ) learns to adaptively route information through finetuned or pre-trained layers . Conditionally parameterized convolutions ( Yang et al. , 2019 ) or dynamic convolutions ( Chen et al. , 2019a ; Verelst & Tuytelaars , 2019 ) have also been proposed to learn specialized convolutional kernels for each example to improve efficiency in image recognition . Our method is also related to recent works on dynamic channel pruning ( Gao et al. , 2018 ; Lin et al. , 2017 ) that generate decisions to skip the computation for a subset of output channels . While GaterNet ( Chen et al. , 2019b ) proposes a separate gating network to learn channel-wise binary gates for the backbone network , Channel gating network ( Hua et al. , 2019 ) identifies regions in the features that contribute less to the classification result , and skips the computation on a subset of the input channels for these ineffective regions . In contrast to the prior works that focus on only dropping unimportant channels , our proposed approach also reuses history features when necessary to make the network capable for strong temporal modelling . 3 METHODOLOGY . In this section , we first show the general approach using 2D-CNN for action recognition . Then we present the concept of adaptive temporal fusion and analyze its computation cost . Finally , we describe the end-to-end optimization and network specifications . Using 2D-CNN for Action Recognition . One popular solution is to first generate frame-wise predictions and then utilize a consensus operation to get the final prediction ( Wang et al. , 2016 ) . The network takes uniformly sampled T frames { X1 ... XT } and predicts the un-normalized class score : P ( X1 , ... , XT ; Θ ) = G ( F ( X1 ; Θ ) , F ( X2 ; Θ ) , ... , F ( XT ; Θ ) ) ( 1 ) where F ( · ; Θ ) is the 2D-CNN with learnable parameters Θ . The consensus function G reduces the frame-level predictions to a final prediction . One common practice for G is the averaging operation . The major drawback is that this can not capture the order of the frames . The network performs poorly on datasets that contain temporal-related labels ( e.g . “ turning left ” , “ moving forward ” , etc ) . LSTM ( Hochreiter & Schmidhuber , 1997 ) can also be used as G to get the final prediction ( Donahue et al. , 2015 ) , but it can not capture low-level features across the frames , as mentioned in Lin et al . ( 2019 ) . A few works have been recently proposed to model temporal causality using a bag of features scheme in TRN ( Zhou et al. , 2018 ) , channel shifting in TSM ( Lin et al. , 2019 ) , depthwise convolutions in TAM ( Fan et al. , 2019 ) . Different from these methods , in this work , we hypothesis that an inputdependent fusion of framewise features will be beneficial for temporal understanding and efficiency , as the amount of temporal information depends on the dynamics and the type of events in the video . Hence we propose adaptive temporal fusion for action recognition . Adaptive Temporal Fusion . Consider a single 2D convolutional layer : yt = φ ( Wx ∗xt+bx ) , where xt ∈ Rc×h×w denotes the input feature map at time step t with c channels and spatial dimension h × w , and yt ∈ Rc ′×h′×w′ is the output feature map . Wx ∈ Rc ′×k×k×c denotes the convolution filters ( with kernel size k× k ) and bx ∈ Rc ′ is the bias . We use “ ∗ ” for convolution operation . φ ( · ) is the combination of batchnorm and non-linear functions ( e.g . ReLU ( Nair & Hinton , 2010 ) ) . We introduce a policy network consisting of two fully-connected layers and a ReLU function designed to adaptively select channels for keeping , reusing or skipping . As shown in Figure 1 , at time t , we first generate feature vectors vt−1 , vt ∈ Rc from history feature map xt−1 and current feature map xt via global average pooling . Then the policy network predicts : pt = g ( vt−1 , vt ; Θg ) ( 2 ) where pt ∈ { 0 , 1 , 2 } c ′ is a channel-wise policy ( choosing “ keep ” , “ reuse ” or “ skip ” ) to generate the output feature map : if pit = 0 , the i-th channel of output feature map will be computed via the normal convolution ; if pit = 1 , it will reuse the i-th channel of the feature map yt−1 which has been already computed at time t − 1 ; otherwise , the i-th channel will be just padded with zeros . Formally , this output feature map can be written as ỹt = f ( yt−1 , yt , pt ) where the i-th channel is : ỹit = 1 [ pit = 0 ] · yit + 1 [ pit = 1 ] · yit−1 ( 3 ) here 1 [ · ] is the indicator function . In Figure 1 , the policy network instructs the convolution layer to only compute the first and fourth channels , reuses the second channel of the history feature and skips the third channel . Features from varied time steps are adaptively fused along the channel dimension . Adaptive temporal fusion enables the 2D convolution to capture temporal information : its temporal perceptive field grows linearly to the depth of the layers , as more features from different time steps are fused when going deeper in the network . Our novel design can be seen as a general methodology for many state-of-the-art 2D-CNN approaches : if we discard `` skip '' and use a predefined fixed policy , then it becomes the online temporal fusion in Lin et al . ( 2019 ) . If the policy only chooses from `` skip '' and `` keep '' , then it becomes dynamic pruning methods ( Gao et al. , 2018 ; Hua et al. , 2019 ) . Our design is a generalized approach taking both temporal modelling and efficiency into consideration . Complexity Analysis . To illustrate the efficiency of our framework , we compute the floating point operations ( FLOPS ) , which is a hardware-independent metric and widely used in the field of efficient action recognition1 ( Wu et al. , 2019b ; Gao et al. , 2020 ; Meng et al. , 2020 ; Fan et al. , 2019 ) . To compute saving from layers before and after the policy network , we add another convolution after ỹt with kernel Wy ∈ Rc ′′×k′×k′×c′ and bias by ∈ Rc ′′ . The total FLOPS for each convolution will be : { mx = c ′ · h′ · w′ · ( k · k · c+ 1 ) my = c ′′ · h′′ · w′′ · ( k′ · k′ · c′ + 1 ) ( 4 ) When the policy is applied , only those output channels used in time t or going to be reused in time t+ 1 need to be computed in the first convolution layer , and only the channels not skipped in time t count for input feature maps for the second convolution layer . Hence the overall FLOPS is : M = T−1∑ τ=0 [ 1 c′ c′−1∑ i=0 Keep at τ or resue at τ + 1︷ ︸︸ ︷ 1 [ piτ · ( piτ+1 − 1 ) = 0 ] ·mx︸ ︷︷ ︸ FLOPS from the first conv at time τ + ( 1− 1 c′ c′−1∑ i=0 Skip at τ︷ ︸︸ ︷ 1 ( piτ = 2 ) ) ·my︸ ︷︷ ︸ FLOPS from the second conv at time τ ] ( 5 ) Thus when the policy network skips more channels or reuses channels that are already computed in the previous time step , the FLOPS for those two convolution layers can be reduced proportionally . Loss functions . We take the average of framewise predictions as the video prediction and minimize : L = ∑ ( x , y ) ∼Dtrain [ −y log ( P ( x ) ) + λ · B−1∑ i=0 Mi ] ( 6 ) 1Latency is another important measure for efficiency , which can be reduced via CUDA optimization for sparse convolution ( Verelst & Tuytelaars , 2019 ) . We leave it for future research . The first term is the cross entropy between one-hot encoded ground truth labels y and predictions P ( x ) . The second term is the FLOPS measure for all the B temporal fusion blocks in the network . In this way , our network is learned to achieve both accuracy and efficiency at a trade-off controlled by λ. Discrete policies for “ keep ” , “ reuse ” or “ skip ” shown in Eq . 3 and Eq . 5 make L non-differentiable hence hard to optimize . One common practice is to use a score function estimator ( e.g . REINFORCE ( Glynn , 1990 ; Williams , 1992 ) ) to avoid backpropagating through categorical samplings , but the high variance of the estimator makes the training slow to converge ( Wu et al. , 2019a ; Jang et al. , 2016 ) . As an alternative , we use Gumbel-Softmax Estimator to enable efficient end-to-end optimization . Training using Gumbel Softmax Estimator . Specifically , the policy network first generates a logit q ∈ R3 for each channel in the output feature map and then we use Softmax to derive a normalized categorical distribution : π = { ri|ri = exp ( qi ) exp ( q0 ) +exp ( q1 ) +exp ( q2 ) } . With the Gumbel-Max trick , discrete samples from the distribution π can be drawn as ( Jang et al. , 2016 ) : r̂ = argmaxi ( log ri+Gi ) , where Gi = − log ( − logUi ) is a standard Gumbel distribution with i.i.d . Ui sampled from a uniform distribution Unif ( 0 , 1 ) . Since the argmax operator is not differentiable , the Gumbel Softmax distribution is used as a continuous approximation . In forward pass we represent the discrete sample r̂ as a one-hot encoded vector and in back-propagation we relax it to a real-valued vector R = { R0 , R1 , R2 } via Softmax as follows : Ri = exp ( ( log ri +Gi ) /τ ) ∑2 j=1 exp ( ( log rj +Gj ) /τ ) ( 7 ) where τ is a temperature factor controlling the “ smooothness ” of the distribution : lim τ→∞ R converges to a uniform distribution and lim τ→0 R becomes a one-hot vector . We set τ = 0.67 during the training . Network Architectures and Notations . Our adaptive temporal fusion module can be easily plugged into any existing 2D-CNN models . Specifically , we focus on BN-Inception ( Ioffe & Szegedy , 2015 ) , ResNet ( He et al. , 2016 ) and EfficientNet ( Tan & Le , 2019 ) . For Bn-Inception , we add a policy network between every two consecutive Inception modules . For ResNet/EfficientNet , we insert the policy network between the first and the second convolution layers in each “ residual block '' / “ inverted residual block '' . We denote our model as AdaFuseMethodBackbone , where the “ Backbone ” is chosen from { “ R18 ” ( ResNet18 ) , “ R50 ” ( ResNet50 ) , “ Inc ” ( BN-Inception ) , “ Eff ” ( EfficientNet ) } , and the “ Method ” can be { “ TSN ” , “ TSM ” , “ TSM+Last ” } . More details can be found in the following section .

Summary: The paper presented an adaptive inference model for efficient action recognition in videos. The core of the model is the dynamic gating of feature channels that controls the fusion between two frame features, whereby the gating is conditioned on the input video and helps to reduce the computational cost at runtime. The proposed model was evaluated on several video action datasets and compared against a number of existing deep models. The results demonstrated a good efficiency-accuracy trade-off for the proposed model.