AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognition
Authors: Yue Meng
Rameswar Panda
Chung-Ching Lin
Prasanna Sattigeri
Leonid Karlinsky
Kate Saenko
Aude Oliva
Rogerio Feris
Venue: ICLR 2021 Poster (2021)

Abstract: Temporal modelling is the key for efficient video action recognition. While understanding temporal information can improve recognition accuracy for dynamic actions, removing temporal redundancy and reusing past features can significantly save computation leading to efficient action recognition. In this paper, we introduce an adaptive temporal fusion network, called AdaFuse, that dynamically fuses channels from current and past feature maps for strong temporal modelling. Specifically, the necessary information from the historical convolution feature maps is fused with current pruned feature maps with the goal of improving both recognition accuracy and efficiency. In addition, we use a skipping operation to further reduce the computation cost of action recognition. Extensive experiments on SomethingV1 & V2, Jester and Mini-Kinetics show that our approach can achieve about 40% computation savings with comparable accuracy to state-of-the-art methods. The project page can be found at https://mengyuest.github.io/AdaFuse/

Published as a conference paper at ICLR 2021
ADAFUSE: ADAPTIVE TEMPORAL FUSION NETWORK FOR EFFICIENT ACTION RECOGNITION
Yue Meng' Rameswar Panda
Chung-Ching Lin
Prasanna Sattigeri
Leonid Karlinsky
Kate S
Aude Oliva 1.2
Rogerio Feris
I "Massachusetts Institute of Technology
MITTIBM Watson Al Lab
3 IBM Research
*Microsoft "Boston University
ABSTRACT
Temporal modelling is the key for efficient video action recognition. While un- derstanding temporal information can improve recognition accuracy for dynamic actions, removing temporal redundancy and reusing past features can significantly save computation leading to efficient action recognition. In this paper, we introduce an adaptive temporal fusion network, called AdaFuse, that dynamically fuses chan- nels from current and past feature maps for strong temporal modelling. Specifically, the necessary information from the historical convolution feature maps is fused with current pruned feature maps with the goal of improving both recognition accuracy and efficiency. In addition, we use skipping operation to further reduce the com- putation cost of action recognition. Extensive experiments on Something &I&V2, V2, Jester and Mini-Kinetics show that our approach can achieve about 40% compu- tation savings with comparable accuracy to maaeeoof-hee tee-the-rt art methods. The project page can be found at https: : :ttee//mmmenne //mengyuest. g thub. 1o/AdaFa AddaFus//
INTRODUCTION
Over the last few years, video action recognition has made rapid progress with the introduction of number of large-scale video datasets (Carreira & Zisserman, 2017: Monfort et al., 2018; Goyal et al., 2017). Despite impressive results on commonly used benchmark datasets, efficiency remains great challenge for many resource constrained applications due to the heavy computational burden of deep Convolutional Neural Network (CNN) models.
Motivated by the need of efficiency, extensive studies have been recently conducted that focus on either designing new lightweight architectures (e.g., R(2+ 1)D (Tran et al., 2018), S3D (Xie et al., CNNs (Tran et al., 2019)) or selecting salient frames/clips conditioned on the input (Yeung al., 2016; Wu al., 2019b; Korbar et al., 2019; Gao al., 2020). However, most of the existing approaches do not consider the fact that there exists redundancy in CNN features which can significantly save computation leading to more efficient action recognition. orthogonal to the design of compact models, the computational cost of a CNN model also has much to do with the redundancy of CNÑ features (Han et al., 2019). Furthermore, the amount of redundancy depends on the dynamics and type of events in the video: set of still frames for simple action (e.g. "Sleeping") will have higher redundancy comparing to a fast-changed action with rich interaction and deformation (c.g. "Pulling two ends of something so that gets stretched"). Thus, based on the input we could compute just a subset of features, while the rest of the channels can reuse history feature maps or even be skipped without losing any accuracy, resulting in large computational savings compared to computing all the features at a given CNN layer. Based on this intuition, we present new perspective for efficient action recognition by adaptively deciding what channels to compute or reuse, per instance basis, for recognizing complex actions.
2018), 1), channel-separateee
particular,
In this paper, we propose AdaFuse, an adaptive temporal fusion network that learns a decision policy to dynamically fuse channels from current and history feature maps for efficient action recognition. Specifically, our approach reuses history features when necessary (i.c., dynamically decides which channels to keep, reuse skip per layer and per instance) with the goal of improving both recognition
"Email: mengyuethu@gmai com. This work was done while Yue was AT Resident IBM Research.

---

Published as a conference paper at ICLR 2021
accuracy and efficiency. As these decisions are discrete and non-differentiable, rely Gumbel Softmax sampling approach (Jang et al., 2016) to learn the policy jointly with the network parameters through standard propppaop,,,iion. without resorting to complex reinforcement learning in (Wu et al., 2019b; Fan et al., 2018; Yeung et al., 2016). We design the loss to achieve both competitive performance and resource efficiency required for action recognition. Extensive experiments on multiple benchmarks show that AdaFuse significantly reduces the computation without accuracy loss.
The main contributions of our work are as follows:
We propose novel approach that automatically determines which channels to keep, reuse or skip per layer and per target instance for efficient action recognition.
Our approach is range of 2D CNN-based action recognition architectures.
which allows this to be served as plugin operation for wide
The overall policy distribution can be seen as an indicator for the dataset characteristic, and the block-level distribution can bring potential guidance for future architecture designs.
We conduct extensive experiments on four benchmark datasets Sranething-Somettiing V1 (Goyal al., 2017), Something-Something V2 (Mahdisoltani et al., 2018), Jester (Materzynska et al., 2019) and Mini-Kinetics (Kay et al., 2017)) to demonstrate the superiority of our proposed approach over state-of-the-ee methods.
2 RELATED WORK
Action Recognition. Much progress has been made in developing variety of ways to recognize complex actions, by either applying 2D-CNNs (Karpathy et al., 2014; Wang et al., 2016; Fan et al., 2019) or 3D-CNNs (Tran et al., 2015; Carreira & Zisserman. 2017; Hara et al., 2018). Most successful architectures are usually based on the two-stream model (Simonyan & Zisserman, 2014), processing RGB frames and optical-flow in two separate CNNs with a late fusion in the upper layers (Karpathy et al., 2014) or further combining with other modalities (Asghari-Esfidden et al., 2020; Li et al., 2020a). Another popular approach for CNN- based action recognition is the use of 2D-CNN to extract frame-level features and then model the temporal causality using different aggregation modules such as temporal averaging in TSN (Wang al., 2016), bag of features scheme in TRN (Zhou al., 2018), channel shifting in TSM (Lin et al., 2019), depthwise convolutions in TAM (Fan et al., 2019). non-local neural networks (Wang et al., 201 20), temporal enhancement and interaction module in TEINet (Liu et al., 2020), and LSTMs (Donahue et al., 2015). Many variants of 3D-CNNs such as C3D (Tran et al., 2015; Ji al., 2013), I3D (Carreira & Zisserman, 2017) and ResNet31) (Hara et al., 201 8), that use 3D convolutions to model space and time jointly, have also been introduced for action recognition. SlowFast (Feichtenhofer et al., 2018) employs two pathways to capture temporal information by processing video at both slow and fast frame rates. Recently, STM (Jiang et al., 2019) proposes new channel-wise convolutional blocks to jointly capture spatio-tempore and motion information in consecutive frames. TEA (Li et al., 2020b) introduces motion excitation module including multiple temporal aggregation modules to capture both short- and long-range temporal evolution in videos. Gate-Shift networks (Sudhakaran et al., 2020) use spatial gating for spatial-temporal decomposition of 3D kernels in Inception- based architectures.
While extensive studies have been conducted the last few years, limited efforts have been made towards efficient action recognition (Wu et al., 90;9b;a; Gao et al., 2020). Specifically, methods for efficient recognition focus on either designing new lightweight architectures that aim reduce the complexity by decomposing the 3D convolution into 2D spatial convolution and 1D temporal convolution (e.g., R(2+1)D (Tran et al., 2018), S3D (Xie et al., 2018), separaeeee CNNs (Tran et al., 2019)) or selecting salient frames/clips conditioned on the input (Yeung et al., 2016; Wu et al., 2019b; Korbar et al., 2019; Gao et al., 2020). Our approach most related to the latter which focuses conditional computation and is agnostic to the network architecture used for recognizing actions. However, instead of focusing on data sampling, our approach dynamically fuses channels from current and history feature maps to reduce the computation. Furthermore, feature maps can be redundant or noisy, we use a skipping operation to make it more efficient for action recognition.
Conditional Computation. Many conditional computation methods have been recently proposed with the goal of improving computational efficiency (Bengio et al., 2015; 2013; Veit & Belongie, 2018; Wang al., 2018 8b; Graves, 2016; Meng et al., 2020; Pan et al., 2021). Several works have been
2

---

Published as a conference paper at ICLR 2021
Keep
Reuse
Skip
2D Conv. (We.
Vt-
Xt-1
Yt-1
Policy Net
AvgPool
Fuse
w
Pt
Vt
2D Conv.
(Wg, bz)
Xt
Vt
yt
Figure 1: conceptual view for adaptive temporal fusion. At time the 2D Conv layer computes for those "keep" channels (blue) in feature map It, and fuses the "reuse" channels (yellow) from the history feature map 1/-1-- The downstream 2D Conv layer (not shown here) will process those "reuse" and "keep" channels in Dt. Best viewed in color.
proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference (Figurnov et al., 2017; McGill Perona, 2017: Wu al., 2020). BlockDrop (Wu et al., 2018) 8) effectively reduces the inference time by learning to dynamically select which layers to execute per sample during inference. SpotTune (Guo al., 2019) learns adaptively route information through finetuned or pre-trained layers. Conditionally parameterized convolutions (Yang al., 2019) or dynamic convolutions (Chen et al., 2019a; Verelst & Tuytelaars, 2019) have also been proposed to learn specialized convolutional kernels for each example to improve efficiency in image recognition. Our method also related to recent works on dynamic channel pruning (Gao et al., 2018; Lin et al., 2017) that generate decisions to skip the computation for subset of output channels. While GaterNet (Chen et al., 2019b) proposes a separate gating network to learn channel-wise binary gates for the backbone network, Channel gating network (Hua et al., 2019) identifies regions in the features that contribute less to the classification result, and skips the computation on subset of the input channels for these ineffective regions. In contrast the prior works that focus only dropping unimportant channels, our proposed approach also reuses history features when necessary to make the network capable for strong temporal modelling.
METHODOLOOGY
In this section, we first show the general approach using 2D-CNN for action recognition. Then we present the concept of adaptive temporal fusion and analyze its computation cost. Finally, descrihe the end-to-end optimization and network specifications.
Using 2D-CNN for Action Recognition. One popular solution is first generate frame-wise predictions and then utilize consensus operation to get the final prediction (Wang et al., 2016). The network takes uniformly sampled T frames {XI... and predicts the un-normalized class score:
-X., - (F (( F(XT:;)
(1)
where F(:; Q) is the 2D-CNN with learnable parameters O. The consensus function reduces the frame-level predictions to a final prediction. One common practice for G is the averaging operation.
The major drawback is that this cannot capture the order of the frames. The network performs poorly on datasets that contain temporal-related labels (c.g. "turning left", "moving forward". etc). LSTM (Hochreiter Schmidhuber, 1997) can also be used as G to get the final prediction (Donahue et al., 2015), but it cannot capture low-level features across the frames, as mentioned in Lin et al. (2019). A few works have been recently proposed to model temporal causality using a bag features scheme in TRN (Zhou et al., 2018), channel shifting in TSM (Lin et al., 2019), depthwise convolutions in TAM (Fan et al., 2019). Different from these methods, in this work, we hypothesis that an input- dependent fusion of framewise features will be beneficial for temporal understanding and efficiency, as the amount temporal information depends on the dynamics and the type of events in the video. Hence propose adaptive temporal fusion for action recognition.
3

---

Published as a conference paper at ICLR 2021
Adaptive Temporal Fusion. Consider single 2D convolutional layer: y/2 <( * +b2), where Iy E Roxhx denotes the input feature map at time step t with с channels and spatial dimension x w, and y E R x xw" is the output feature map. Wa E Re'xkxxxe denotes the convolution filters (with kernel size k x k) and by E R is the bias. We use "*" for convolution operation. Φ(-) is the combination of batchnorm and non-linear functions (e.g. ReLU (Nair & Hinton, 2010)).
We introduce policy network consisting of two fully-connected layers and a ReL.U function designed to adaptively select channels for keeping, reusing or skipping. As shown in Figure 1, at time we first generate feature vectors Ut-1, 1t E I from history feature map 2 and current feature map 2t via global average pooling. Then the policy network predicts:
Pr = g(t fg)
(2)
where p: E {0, 1, 2) is channel-wise policy (choosing "keep", "reuse" or "skip") to generate the output feature map: if p) 0, the i-th channel of output feature map will be computed via the normal convolution; if p = 1, it will reuse the i-th channel of the feature map yt-1 which has been already computed time 1: otherwise, the i-th channel will be just padded with zeros. Formally, this output feature map can be written as T/+ = fyt-11 Dt.Pt) where the i-th channel is:
p 0] vi 1 =
here 1 [.] is the indicator function. In Figure 1, the policy network instructs the convolution layer to only compute the first and fourth channels, reuses the second channel of the history feature and skips the third channel. Features from varied time steps are adaptively fused along the channel dimension.
Adaptive temporal fusion enables the 2D convolution to capture temporal information: its temporal perceptive field grows linearly to the depth of the layers, more features from different time steps are fused when going deeper in the network. Our novel design can be seen as a general methodology for many state-of-the-an 2D-CNN approaches: if we discard "skip" and use predefined fixed policy, then it becomes the online temporal fusion in Lin et al. (2019). If the policy only chooses from "skip" and "keep", then becomes dynamic pruning methods (Gao et al., 2018; Hua et al., 2019). Our design is generalized approach taking both temporal modelling and efficiency into consideration.
Complexity Analysis. To illustrate the efficiency of our framework, we compute the floating point operations (FLOPS), which is hardwarr-inddependet metric and widely used in the field of efficient action recognition (Wu et al., 2019b; Gao et al., 2020; Meng et al., 2020; Fan et al., 2019). To compute saving from layers before and after the policy network, we add another convolution after jt with kernel Wy E Re xk' xk xe' and bias by E R.". The total FLOPS for each convolution will be:
(m= = c' W' w/ (k. c m.
my my = d" h² wn .K' K 1)
When the policy is applied, only those output channels used in time or going to be reused in time t++ need to be computed in the first convolution layer, and only the channels not skipped in time count for input feature maps for the second convolution layer. Hence the overall FLOPS is:
Keep resue
Skip at
T-1
M =
Σ 7=0
1 pt (r/+1 - = 0 m. + Σ 2=0
1pp²= = my
(5)
1=0
FLOPS from the first conv time 7
FLOPS from the second coav time
Thus when the policy network skips more channels or reuses channels that are already computed in the previous time step, the FLOPSS for those two convolution layens can be reduced proportionally.
Loss functions. We take the average of framewise predictions as the video prediction and minimize:
B B-1
L=
Σ
- yylog(((x)) + +
Σ
M₁
(6)
(z.g)>-Deram
1=
II 'Laenny is another important measure for efficiency, which can be reduced via CUDA optimization for sparse convolution (Verelst & Tuytelaars, 2019). We leave for future research.
4

---

Published as a conference paper at ICLR 2021
The first term is the cross entropy between one-hot encoded ground truth labels y and predictions P(x). The second term is the FLOPS measure for all the B temporal fusion blocks in the network. In this way, network is learned to achieve both accuracy and efficiency at trade-off controlled by X.
Discrete policies for "keep". "reuse" or "skip" shown in Eq. and Eq. 5 make L non-diffcrentiable hence hard to optimize. One common practice is to use score function estimator (e.g. REINFORCE (Glynn, 1990; Williams, 1992)) to avoid backpropagating through categorical samplings, but the high variance of the estimator makes the training slow to converge (Wu et al., 2019a; Jang et al., 2016). As an alternative, we use Gumbel-Softmax Estimator to enable efficient end-to-end optimization.
Training using Gumbel Softmax Estimator. Specifically, the policy network first generates a logit
E R23 for each channel in the output feature map and then we use Softmax to derive a normalized categorical distribution:
{ralr: exp +cxp (91))()))+ex)) texp [og] ). With the Gumbel-Max trick, discrete samples from the distribution ıT can be drawn as (Jang et al., 2016): f = argmax, (log where G; = log(- log U,) is a standard Gumbel distribution with i.i.d. U; sampled from uniform distribution Un 1). Since the argmax operator is not differentiable, the Gumbel Softmax distribution is used as continuous approximation. In forward pass represent the discrete sample f as a one- hot encoded vector and in back-propagation we relax it to a real-valued vector R { [o; R1, ] via Softmax follows:
exp(a,
r.+ r. G,),
exp (log ri + Gt)/T)
R4
ΣE= cxp (logrr, Gz)/T)
where r is a temperature factor controlling the "smooothness" of the distribution: lim R converges T-30
to uniform distribution and lim becomes a one-hot vector. We set r 0.67 during the training.
=-0 Network Architectures and Notations. Our adaptive temporal fusion module be easily plugged into any existing 2D-CNN models. Specifically, we focus on BN-Inception (Ioffe & Szegedy, 2015), ResNet (He et al., 2016) and EfficientNet (Tan & Le, 2019). For Bn-Inception, we add policy network between every two consecutive Inception modules. For ResNet/EffiiienNNe, we insert the policy network between the first and the second convolution layers in each "residual "lock"/"inverrte" residual block". We denote our model as Metee ("R18 ("R18"(Res) (Ressee))))))) (ResNet05),,, "Inc"(BN- (BN-Inceptioon) (Eficcitt)))ee and the "Method" can be ("TSN", "TSM", "TSM+Last"). More details can be found in the following section.
Rockmmee where the "Backbone" is chosen from
EXPERIMENTS
We first show AdaFuse can significantly improve the accuracy and efficiency of ResNet18, BN- Inception and EfficientNet. outperforming other baselines by a large margin on Something- V1. Then on all datasets, AdaFuse with ResNet18/ / ResNet50 can consistently outperform corresponding base models. We further propose two instantiations using AdaFuse on TSM (Lin et al., 2019) to compare with state-of-the-an approaches on Something V1 & V2: AdaFuse TSM can save over 40% FLOPS comparable classification score under same amount of computation budget, AdaFuse]TM-laa outperforms state-of the-hartr art methods in accuracy. Finally, we perform comprehensive ablation studies and quantitative analysis to verify the effectiveness of our adaptive temporal fusion.
Datasets. We evaluate AdaFuse on Something-Someetie
VI (Goyal et al., 2017) & V2 (Mahdis- oltani et al., 2018), Jester (Materzynska al., 2019) and a subset of Kineties (Kay et al., 2017). Something V1 (98k videos) & V2 (194k videos) are two large-scale datasets sharing 174 human action labels (e.g. pretend to pick something up). Jester (Materzynska et al., 2019) has 27 annotated (assembled
classes for hand gestures, with 1 119k / 15k videos in training/ / validation set. by Meng et al. (2020) a subset of full Kinetics dataset (Kay et al., 2017) containing 121k videos for training and 10k videos for testing across 200 action classes.
Implementation details. To make a fair comparison, we carefully follow the training procedure in Lin et al. (2019). We uniformly sample T = 8 frames from each video. The input dimension for the network is 224 x 224. Random scaling and cropping are used as data augmentation during training (and we further adopt random flipping for Mini-Kinetion Center cropping is used during inference. All our networks are using ImageNet pretrained weights. We follow a step-wise learning rate scheduler with the initial learning rate 0.002 and decay by 0.1 epochs 20 & 40. To train
5

---

Published as a conference paper at ICLR 2021
Table 1: Action Recognition Results Semethiin--Smmthhngg Dataset. Our proposed method consistently outperforms all other basclines in both accuracy and efficiency.
Method
Backbone
#Params
FLOPS
Top1
Top5
TSN (Wang al., 2016)
ResNet18
11.2M
14.6G
14.8
38.0
TSN
(Wang al., 2016)
BN-Inception
10.4M
16.4G
17.6
43.5
CGNet (Hua ct al., 2019)
ResNet18
11.2M
11.2G
13.7
35.1
Threshold
ResNet18
11.2M
11.3G
14.1
36.6
Random
ResNet18
11.2M
10.4G
27.5
54.2
LSTM
ResNet18
11.7M
14.7G
28.4
56.3
AdaFuse
ResNet18
15.6M
10.3G
36.9
65.0
AdaFuse ISN
BN- BN-Inception
14.5M
12.1G
38.5
67.8
Table 2:
Action Recognition on Something- V1 using EfficientNet architecture. consistently outperforms all the EfficientNet baselines in both accuracy and efficiency.
AdaFuse
Method
Backbone
#Params
FLOPS
Top1
Top5
TSN
EfficientNet-hO
5.3M
3.1G
18.0
44.9
TSN
EfficientNet-hl
7.8M
5.6G
19.3
45.9
TSN
EfficientNet-b2
9.2M
8.0G
18.8
46.0
TSN
EfficientNNttb3
12.0M
14.4G
19.3
46.6
AdaFuse TSN
EfficientNet-b0
9.3M
2.8G
39.0
68.1
AdaFuse ISN
EfficientNet-bl
12.4M
4,9G
40.3
69.2
AdaFuse TSM
EfficientNet-B
13.8M
7.2G
40.2
69.5
AdaFuse
Efficienteetb33 16.6M
12.9G
40.7
69.7
BIT-3
our adaptive temporal fusion approach, we set the efficiency term À 0.1. We train all the models for 50 epochs with a batch-size of 64, where each experiment takes 12~ 24 hours on 4 Tesla V100 GPUs. We report the number of parameters used in each method, and measure the averaged FLOPS and Top5 accuracy for all the samples from each testing dataset.
Adaptive Temporal Fusion improves 2D CNN Performance. On Something V1 dataset, show AdaFuse 's improvement upon 2D CNNs by comparing with several baselines as follows:
TSN (Wang al., 2016): Simply average frame-level predictions the video-level prediction.
CGNet (Hua al., 2019): A dynamic pruning method reduce computation cost for CNNs.
Threshold: We keep fixed portion of channels base their activation L1 norms and skip the channels smaller norms. It serves as a baseline for efficient recognition.
RANDOM: We use temporal fusion with randomly sampled policy (instead of using learned policy distribution). The distribution is chosen to match the FLOPS of adaptive methods.
Update per-frame predictions by hidden states in LSTM and averages all predictions as the video-level prediction.
LSTM:
We implement all the methods using publicly available code and apply adaptive temporal fusion in TSN using ResNet18, BN-Inception and EfficientNet backbones, denoting them as AdaFuse AdaFuse/SS and AdaFuse!IS respectively ("x"" stands for different scales of the EfficientNet back- bones). As shown in Table AdaFusezyy uses the similar FLOPS as those efficient methods ("CGNet" and "Threshold") but has great improvement classification accuracy Specifically, AdaFuse and AdaFuse/3N outperform corresponding TSN models by more than 20% in Top-1 accuracy, while using only 74% of FLOPS. Interestingly, comparing to TSN, even temporal fusion with a random policy can achieve an absolute gain of 12.7% in accuracy, which shows that temporal fusion can greatly improve the action recognition performance 2D CNNs. Additionally equipped with the adaptive policy, AdaFuse]SN can get 9.4% extra improvement in classification. LSTM is the most competitive baseline in terms accuracy, while AdaFuse gis has an absolute gain of 8.5% in accuracy and uses only 70% of FLOPS. When using still reduce 10% of the FLOPS while improving the accuracy by a large margin. To further validate AdaFuse being madel-agnostic and robust, conduct extensive experiments using ResNet18 and
efficient architecture shown in Table.2,
approach

---

Published as a conference paper at ICLR 2021
Table 3: Comparison with TSN using ResNet-18/Res backbones.
AdaFuse consistently outperforms TSN by large margin in accuracy while offering significant savings in FLOPs.
Something FLOPS
Something V2
Jester
Mini-Kinetics
Method
#Params
Topl
FLOPS
Top1
FLOPS
Topl
FLOPS
Topl
TSNR18 (Wang et al., 2016)
11.2M
14.6G
14.8
14.6G
27.3
14.6G
82.6
14.6G
64.6
LST
11.7M
14.7G
28.4
14.7G
40.3
14.7G
93.5
14.7G
67.2
AdaFuse RI8
TSN
15.6M
10.3G
36.9
11.1G
50.5
7.6G
93.7
11.8G
67.5
TSNs50 (Wang et al., 2016)
23.6M
32.9G
18.7
32.9G
32.1
32.9G
82.6
32.9G
72.1
MS
28.8M
33.0G
30.1
33.0G
47.4
33.0G
93.7
33.0G
71.6
AdaFusel30
37.8M
22.1G
41.9
18.1G
56.8
16.1G
94.7
23.0G
72.3
Table 4: Comparison with the recent adaptive inference method AR-Net (Meng et al., 2020) on Sewething-Sometingg--ye Jester and Mini-Kinetics datasets, AdaFuse achieves a better accuracy with great savings in computation (FLOPS) and number of parameters.
SomethingVi FLOPS Top1
Jester FLOPS
Mini-Kinetics FLOPS
Method
#Params
Topl
Top1
AR-Net (Meng et al., 2020)
63.0M
41.4G
18.9
21.2G
87.8
32.0G 23.0G
71.7
Ad
37.8M
22.1G
41.9
16.1G
94.7
72.3
Table 5: Comparison with
methods Our method has comparative accuracy with great savings in FLOPS.
V1 & V2 datasets.
#Params Something- V1
Something- V2
Method
Backbone
T
FLOPS
Top1
FLOPS
Top1
TSN (Wang et al., 2016)
BN-Inception
8
10.7M
16.0G
19.5
16.0G
33.4
TSN (Wang al., 2016)
ResNet50
8
24.3M
33.2G
19.7
33.2G
27.8
TRN Muliicall (Zhou et al., 2018)
BN-Inception
8
18.3M
16.0G
34.4
16.0G
48.8
TR FR (Zhou et al., 2018)
BN- IN-Inception
8+8
36.6M
32.0G
42.0
32.0G
55.5
I3D (Carreira & Zisserman. 2017)
3DResNet50
32x2
28.0M
306G
41.6
13D+GCN+NL (Wang & Gupta, 2018)
3DResNet50
32x2
62.2M
606G
46.1
ECO (Zolfaghari al., 2018)
8
47.5M
32G
39.6
Lie (Zolfaghari et al., 2018)
BNinc+3DRes18
92
150M
267G
46.4
TSM (Lin et al., 2019)
ResNet50
8
24.3M
33.2G
45.6
33.2G
59.1
AdaFuse TSN
BN-Inception
8
14.5M
12.1G
38.5
12.5G
53.4
Inc
AdaFuseTSh R50
ResNet50
8
37.7M
22.1G
41.9
18.1G
56.8
AdaFuse R50
ResNet50
8
37.7M
19.1G
44.9
19.5G
58.3
AdaFuseTsMi TMM-Las
ResNet50
8
39.1M
31.5G
46.8
31.3G
59.8
ResNet50 backbones on Something V1 & V2, Jester and Mini-Kinetics As shown in Table 3, AdaFuse and AdaFuse consistently outperform their baseline TSN and LSTM models with 35% saving in FLOPS on average. Our approach harvests large gains in accuracy and efficiency on temporal-rich datasets like Something VI & V2 and Jester. When comes to Mini- Mini-Kinetics, AdaFuse still achieve better accuracy with 33%~ computation reduction.
Comparison with Adaptive Inference Method. We compare our approach with AR-Net (Meng al., 2020), which adaptively chooses frame resolutions for efficient inference. As shown in Table 4, on Something V1, Jester and Mini-Kineni Kinetiss, we achieve a better accuracy efffracyeffiiiency trade-off than AR-Net while using 40% less parameters. On temporal-rich dataset like Something- V1, our approach attains the largest improvement, which shows AdaFuse capability for strong temporal modelling,
Comparison with State-of-the-Art Methods. We apply adaptive temporal fusion with differ- ent backbones (ResNet50 (He et al., 2016), BN-Inception (loffe & Szegedy, 2015)) and designs (TSN (Wang et al., 2016), TSM (Lin al., 2019)) and compare with State-of-the-Art methods on Something V1 & V2. As shown in Table 5, using BN-Inception as backbone, AdaFuseTsN 4% better than "TRNMutiscate" (Zhou al., 2018) in accuracy, using only 75% of the FLOPS. AdaFuseTsy with ResNet50 can even outperform 3D CNN method "I3D" (Carreira & Zisserman, 2017) and hybrid 2D/3D CNN method "ECO" (Zolfaghari et al., 2018) with much less FLOPS.
7

---

Published as a conference paper at ICLR 2021
As for adaptive temporal fusion on "TSM" (Lin et al., 2019), AdaFuse achieves more than 40% savings in computation but at 1% loss in accuracy (Table 5). We believe this is because TSM uses temporal shift operation, which can be seen as a variant of temporal fusion. Too much temporal fusion could cause performance degradation due to worse spatial modelling capability, As remedy, we just adopt adap-
50
STM
ECO
13D+GCN+NL
AdaFuseio TSM
AdaFuse Tit Laxt
TSM
AdaFuse TRNuue How
I3D
ECO
40
AdaFusthu
Accuracy (%)
35
TRNeutiscrre
tive temporal fusion in the last block in TSM to capture high-level semantics (more intuition can be found later in our visualization experiments) and denote it as AdaFuse Lat, On Something VI & V2 datasets, AdaFuseTSMelaw outperforms TSM and all other state-of-the accuracy with a 5% saving in FLOPS comparing to TSM. From our experiments, we observe that the performance of adaptive temporal fusion de- pends on the position of shift modules in TSM and optimizing the position of such modules through additional regularization could help us not only achieve better accuracy but also to lower the number of parameters. We leave this as an interesting future work.
30
25-
methods in
20
TSN
AR Net
10
20
40
30
160
320
640
GHI OPS
Figure 2: FLOPS vs Accuracy on Something- V1 Dataset. The diameter of each data point is propor- tional to the total number of parameters. AdaFuse (blue points) achieves the best trade-off at com- parable model size to 2D CNN approaches.
We depict the accuracy, computation cost and model sizes in Figure 2. All the results are computed from Something V1 validation set. The graph shows GFLOPS accuracy x y-axis and the diameter of each data point is proportional to the number of model parameters. AdaFuse (blue points) owns the best trade-off for accuracy and efficiency at comparable model size to other 2D CNN approaches. Once again it shows AdaFuse is an effective and efficient design for action recognition.
Policy Visualizations. Figure 3 shows overall policy ("Skip", "Reuse" and "Keep") differences across all datasets. We focus on the quotient of "Reuse Keep" as it indicates the mixture ratio for feature fusion. The quotients on Something &1&V2 and Jester datasets very high (0.694, 0.741 and 0.574 respectively) when comparing to Mini-Kinetics (0.232). This is probably because the first three datasets contain more temporal relationship than Kinetics. Moreover, Jester has the highest percentage in skipping which indicates many actions in this dataset can be correctly recognized with few channels: Training on Jester is more biased towards optimizing for efficiency as the accuracy loss is very low. Distinctive policy patterns show different characteristics of datasets, which conveys a potential of our proposed approach to be served as a "dataset inspector".
Figure 4 shows more fine-grained policy distribution on Something V2. We plot the policy usage in each residual block inside the ResNet50 architecture (shown light red/orange/blue) and use 3rd-order polynomials to estimate the trend of each policy (shown in black dash curves). To further study the sime-sensttiveness of the policies, we calculate the number of channels where the policies stay unchanged across the frames in one video (shown in dark red/orange/blue). We find earlier layers tend to skip more and reuse/keep less, and vice versa. The first several convolution blocks normally capture low-level feature maps in large spatial sizes, so the "information density" channel dimension should be less which results in more redundancy across channels. Later blocks often capture high-level semantics and the feature maps are smaller in spatial dimensions, so the "semantic density" could be higher and less channels will be skipped. In addition, low-level features change faster across the frames (shades, lighting intensity) whereas high-level semantics change slowly across the frames (c.g. "kicking soccer"), that's 's why more features can be reused in later layers to avoid computing the same semantic again. As for the time-sensitiveness, earlier layers tend to be less sensitive and vice versa. We find that "reuse" is the most time-sensitive policy, as "Reuse (Instance)" ratio is very low, which again shows the functioning of adaptive temporal fusion. We believe these findings will provide insights to future designs of effective temporal fusions.
How does the adaptive policy affect the performance? We consider AdaFuse VI dataset and break down by using "skip", 'reuse" and adaptive (Ada.) policy learning. As shown
RiB on Something
8

---

Published as a conference paper at ICLR 2021
0.75
Skip
Something-V1
Something-V2
Keep
2.30
SHp
23.2%
025
Skip (inttance)
37.0%
37 2%
Skip
20
Rouse
Rabio
0.2
Reuse
Keep
Minl-Kinetiics
0.00
(instance)
0.30
Keep
Reuse
Ratio
0.20
keep
0.30
Ketp
(instance)
Skip
MOUSE
Bottleneck Block Index
10
Figure 3: speecit- policy distribution.
Figure 4:
Policy distribution and trends for each residual block on Something- V2 dataset.
Table 7: Effect of hidden sizes and efficient weights on the performance of FaaFFusTtMLaa on Sth
Table 6: Effect of different policies (using
A on Something V1 dataset.
Method
Skip Reuse Ada.
FLOPS
Top1
#Hidden Units
À
#Params FLOPS Top1
Skip
Reuse
TSN
x
x
x
14.6G
14.8
Ada. Skip
x
6.6G
9.5
1024
0.050
39. 39.1M
31.53G
59.71
13%
14%
Ada. Reuse
x
V
v
13.8G
36.3
1024
0.075
39.1M
31.29G
59.75
15%
13%
Random
x
10.4G
27.5
1024
0.100
39.1M
31.04G
59.40
18%
12%
AdaFuse K18
10.3G
36.9
2048
0.100
54.3M
30.97G
59.96
21%
10%
4096
0.100
84.7M
31.04G
60.00
25%
8%
in Table 6, "Ada. Skip" saves 55% of FLOPS comparing TSN but at great degradation in accuracy. This shows naively skipping channels won't give a better classification performance. "Ada. Reuse" approach brings 21.5% absolute gain in accuracy, which shows the importance of temporal fusion. However, fails to save much FLOPS due to the absence of skipping operation. Combining "Keep" with both "Skip" and "Reuse" via just random policy is already achieving better trade-off comparing to TSN, and by using adaptive learning approach, AdaFuse reaches the highest accuracy with the second-best efficiency. In summary, the "Skip" operation contributes the most to the computation efficiency, the "Reuse" operation boosts the classification accuracy, while the adaptive policy adds the chemistry the whole system and achieves the best performance.
How to achieve a better performance? Here we investigate different settings to improve the performance of AdaFuse/SMataar on Something V2 dataset. As shown in Table 7, increasing will obtain a better efficiency but might result in accuracy degradation. Enlarging the number of hidden units for the policy network can get better overall performance: as increase the size from 1024 to 4096, the accuracy keeps increasing. When the policy network grows larger, it learns to skip more reduce computations and to reuse history features wisely for recognition. But notice that the model size grows almost linearly to hidden layer sizes, which leads to considerable overhead to the FLOPS computation. As a compromise, we only choose À = 0.75 and hidden size 1024 for AdaFuseeSM--as We leave the design for a more advanced and delicate policy module for future works.
Runtime/Hardware. Sparse convolutional kernels are often less efficient on current hardwares, e.g., GPUs. However, strongly believe that it is important explore models for efficient video action recognition which might guide the direction of new hardware development in the years to come. Furthermore, also expect wall-clock time speed-up in the inference stage via efficient CUDA implementation, which we anticipate will be developed.
5 CONCLUSIONS
We have shown the effectiveness of adaptive temporal fusion for efficient video recognition. Compre- hensive experiments on four challenging and diverse datasets present broad spectrum of accuracy- efficiency models. Our approach is modos-agnostic, which allows to be served as a plugin operation for wide range of architectures for video recognition tasks.

---

Published as a conference paper at ICLR 2021
Acknowledgements. This work is supported by the Intelligence Advanced Research Projects Activity (IARPA) via DOI/IBC contract number D17PC00341. The U.S. Government is authorized to reproduce and distribute reprints for Govermmental purposes notwithstanding any copyright annotation thereon. This work is also partly supported by the MIT-IBM Watson Al Lab.
Disclaimer. The views and conclusions contained herein those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of IARPA, DOI/IBC, or the U.S. Government.
REFERENCES
Sadjad Asghari-Esfeden, Mario Sznaier, and Octavia Camps. Dynamic motion representation for human action recognition. In The IEEE Winter Conference on Applications of Computer Vision, pp. 557-566, 2020.
Emmanuel Bengio, Pierre-Lue Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:/511.0629) 2015.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Yinpeng Chen, Xiyang Dai, Mengchen Liu, Dongdong Chen, Lu Yuan, and Zicheng Liu. Dynamic convolution: Attention over convolution kernels. arXiv preprint
2019a.
Zhourong Chen, Yang Li, Samy Bengio, and Si Si. You look twice: Gaternet for dynamic filter selection in cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9172-9180, 2019b.
Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Mareus Rohrbach, Subhashini Venu- gopalan, Kate Saenko, and Trevor Darrell. Long-tern term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 26344, 2015.
Hehe Fan, Zhongwen Xu, Linchao Zhu, Chenggang Yan, Jianjun Ge, and Yi Yang. Watching a small portion could be good as watching all: Towards efficient video classification. In IJCAI International Joint Conference on Artificial Intelligence, 2018
Quanfu Fan, Chun-Fu Richard Chen, Hilde Kuehne, Marco Pistoia, and David Cox. More is less: Learning efficient video representations by big-little network and depthwise temporal aggregation. In Advances Neural Information Processing Systems, pp. 2261-2270, 2019.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. arXiv preprint 1rXiv:1812.03995 2018
Michael Figurnov, Maxwell D Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov. Spatially adaptive computation time for residual networks. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp.
2017.
Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10457-10467, 2020.
Xitong Gao, Yiren Zhao, Lukasz Dudziak, Robert Mullins, and Cheng-zhong Xu. Dynamic channel pruning: Feature boosting and suppression. arXiv preprint arXiv:1810.05311, 2018.
Peter W Glynn. Likelihood ratio gradient estimation for stochastic systems. Communications of the ACM, (3(10)375-84 1990.
10

---

Published as a conference paper at ICLR 2021
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Frelter-Freita,, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, volume 1, pp. 5, 2017.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv preprint 2016.
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition, pp. 4805 2019.
Kai Han, Yunhe Wang, Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. arXiv preprint arXiv: 2019.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2355-6555, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Weizhe Hua, Yuan Zhou, Christopher M De Sa, Zhiru Zhang, and G Edward Suh. Channel gating neural networks. In Advances in Neural Information Processing Systems, pp. 1884-1894, 2019.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 157 2015.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparamcterizattion with gumbel-softmax. arXiv preprint arXiv: I. 00/ 144, 2016.
S. Ji, W. Xu, M. Yang, and K. Yu. 3d convolutional neural networks for human action recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 35(1):221-231, Jan 2013. doi: 10.1 109/TPAMM..212.555
Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2000-2009, 2019.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725-1732, 2014.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayamarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 01777755009955 2017.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Scsampler: Sampling salient clips from video for efficient action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp.
2019.
Chengxi Li. Yue Meng, Stanley H Chan, and Yi-Ting Chen. Learning 3d-aware egocentric spatial- temporal interaction via graph comvolutional networks. In 2020 IEEE International Conference Robotics and Automation (ICRA), pp. 4418-8424. IEEE, 2020a.
Yan Li, Bin Ji, Xintian Shi, Jianguo Zhang, Bin Kang, and Limin Wang. Tea: Temporal excitation and aggregation for action recognition. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 909-918, 2020b.

---

Published as a conference paper at ICLR 2021
Ji Lin, Yongming Rao, Jiwen Lu, and Jie Zhou. Runtime neural pruning. In Advances in Neural Information Processing Systems, pp. 2181-2191, 2017.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7083-7093, 2019.
Zhaoyang Liu, Donghao Luo, Yabiao Wang, Limin Wang, Ying Tai, Chengjie Wang. Jilin Li, Feiyue Huang, and Tong Lu. Teinet: Towards an efficient architecture for video recognition. In AAAI, pp. 11669-11676, 2020.
Farzaneh Mahdisoltani, Guillaume Berger, Waseem Gharbich, David Fleet, and Roland Memisevic. On the effectiveness of task granularity for transfer learning. arXiv preprint arXiv: /001: 09235, 2018.
Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic. The jester dataset: A large-scale video dataset of human gestures. In Proceedings of the IEEE International Conference Computer Vision Workshops, pp. 0-0, 2019.
Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing in artificial neural networks. In Proceedings of the 34th International Conference on Machine Learning-Volume Volume 70, pp. 2363-2372, 2017.
Yue Meng, Chung- Chinn Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris. Ar-net: Adaptive frame resolution for efficient action recognition. In European Conference on Computer Vision, pp. 86-104. Springer, 2020.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. arXiv preprint arXiv: 18010000110053 2018.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Bowen Pan, Rameswar Panda, Camilo Luciano Fosco, Chung-Ching Lin, Alex J Andonian, Yue Meng, Kate Saenko, Aude Oliva, and Rogerio Feris. Va-red?: Video adaptive redundancy reduction. In International Conference on Learning Representations, 2021.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Neural Information Processing System (NIPS), 2014.
Swathikiran Sudhakaran, Sergio Escalera, and Oswald Lanz. Gate-shift networks for video action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1102-1111, 2020.
Mingxing Tan and Quoc V Le. Efficientne:: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv: 1905. 1194., 2019.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489 148994447, 2015.
Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6450-6459, 2018.
Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5552-5561, 2019.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Proceedings of the European Conference Computer Vision (ECCV), pp. 3-18, 2018.
12

---

Published as a conference paper at ICLR 2021
Thomas Verelst and Tinne Tuytelaars. Dynamic convolutions: Exploiting spatial sparsity for faster inference. arXiv preprint arXiv:1912.03200 2019.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In Proceedings of the European conference on computer vision (ECCV), pp. 399 417, 2018.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7794-7803, 2018a
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 409-424, 2018b.
Ronald J Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-2 1992.
Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, Yi Yang, and Shilei Wen. Dynamic inference: A new approach toward efficient video action recognition. arXiv preprint arXiv:2002..3332, 2020.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8817-888 8826, 2018.
Zuxuan Wu, Caiming Xiong, Yu-Gang Jiang, and Larry Davis. Liteeval: A coarse-to-fine framework for resource efficient video recognition. In Advances in Neural Information Processing Systems, pp. 7778-7787, 2019a.
Zuxuan Wu, Caiming Xiong, Chih-Yao Ma, Richard Sacher, and Larry S Davis. Adaframe: Adaptive frame selection for fast video recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1278-1287, 2019b.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking Spatiotemporal Feature Learning: Speed- Accuracy Trade-offs in Video Classification. The European Conference on Computer Vision (ECCV), September 2018.
Brandon Yang, Gabriel Bender, Quoe V Le. and Jiquan Ngiam. Condconw: Conditionally parameter- ized convolutions for efficient inference. In Advances in Neural Information Processing Systems, pp. 1316, 2019.
Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. End-to-end learning of action detection from frame glimpses in vidcos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2678-2687, 2016.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 803-818, 2018.
Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox Eco: Efficient convolutional network for online video understanding. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 695-712, 2018.
13


Reviews:
[{"review_id": "q0iwglOcbT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this work, the authors introduce an AdaFuse network for efficiency action recognition in videos. Specifically, they design a policy net to decide which channels should be kept, reused or skipped, according to the input features of two adjacent frames.\n\nStrength \n\n1 The paper is written well, and the organization is OK\n\n2 The idea of adaptive temporal fusion is somehow novel and interesting \n\nWeakness\n\n1 How to save computation. I understand the general idea of saving computation, if some channels are reused or skipped. However, in the training phase, the policy net would produce the real-value vector by Eq. (7), instead of the one-hot vector. In other words, the 'keep' entry for each channel is always used during training. Then, I guess computation saving is not claimed for training. It is for testing, right? How to do test? The policy net produces the real-value vector and then you make it as one-hot vector for saving computation?\n\n2 Missing SOTA. Compared with this paper, many recent approaches can achieve a competitive computation with better accuracy. It significantly reduces the potential value of this paper.\n\n*Jiang et al., STM: SpatioTemporal and Motion Encoding for Action Recognition, ICCV 2019\n\n*Li et al., TEA: Temporal Excitation and Aggregation for Action Recognition, CVPR 2020\n\n*Sudhakaran et al., Gate-Shift Networks for Video Action Recognition, CVPR2020\n\n*Liu et al., TEINet: Towards an efficient architecture for video recognition, AAAI 2020\n\n3 Please correct the abstract. The experiments are performed on mini-Kinetics, rather than Kinetics. I indeed suggest that, it would be better to perform the proposed method on Kinetics to further show the effectiveness.", "extra_scores": {}, "extra_text": {}}, {"review_id": "l4yF_JCNBO", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### General\nThis paper proposes an adaptive temporal fusion network called AdaFuse for action recognition, which adaptively removes temporal redundancy and reuses past features for accuracy and efficiency.\nI listed the Pros and Cons I found in the paper below as well as some questions to clarify some of the details.\n\n#### Pros\n1. The idea of learning a decision policy to dynamically determine whether channel-wise features at time $t$ are calculated normally, reused from $t-1$, or skipped, is interesting and reasonable.\n1. The experimental results show that the proposed method achieves good accuracy with reasonable computational budget.\n1. The ablation study in Table 4 reveals that the performance is greatly affected by the policy and it is important to fuse the futures from different frames to captures the temporal dependencies.\n\n#### Cons\n1. The propsoed method is not compared with some of the recent methods such as [1-3] ([4] is optional because the publication date is very close to the ICLR 2021 submission deadline). Especially for Jester and Mini-Kinetics dataset, the proposed method is compared with only TSN, which is old and weak as baseline as it does not incorporate the temporal information.\n1. In Table 3, it seems that the proposed method achieves good accuracy, but I am afraid that it is just because of the strong base network, TSM. Merely adding AdaFuse to TSM indeed saves some computation but degrades the performance as described in the paper. The proposed remedy indeed slightly improves the accuracy but it requires much more parameters compared to the vanilla TSM. Overall, I find it benefitial to use the proposed method on top of simple base networks such as TSN, but the benefit of using the proposed method on top of strong base networks such as TSM may be marginal. Combined with the point 1 above, I am not well convinced of the effectiveness of the proposed method.\n1. Some of the important details are not clear. I would appreciate if the authors could answer the questions I listed below.\n\n#### Questions\n1. Is it necessary to use Gumbel softmax? I think there are two kinds of tricks involved in Gumbel softmax. One is a trick for sampling from a categorical distribution, and the other is a trick for making the opperation differentiable. In my understanding, which may be wrong, the required characteristic for the present method is the latter one, and the sampling from the categorical distribution is not necessarily required. In this case, I think simply using $q$ instead of $\\log{r} + G$ in equation (7) is enough.\n1. Related to the point above, please clarify the type of output (hard or soft) of the policy net. The sentence after equation (2) says the output is integer values (0, 1, or 2), while the sentence before equation (7) says it is a real-valued vector.  \n1. Suppose $p_t^i = 1$ (reuse) and $p_{t-1}^i = 1$ (reuse again). In this case, is $y_t^i$ copied from $y_{t-2}^i$ ? Or is the feature map of $i$-th channel at time $t-1$ calculated on the fly for \"reusing\" at time $t$? In other words, if the policies for a channel is \"reuse\" $n$ consecutive times, does the method take the feature from $n$ frames before? \n\n\n#### Other comments\n1. Figure 1 may be incorrect or misleading. I think $p_t$, the output of the policy net, should go to the 2D Conv. block. Otherwise the block never knows which channel to compute at time $t$ and which channel to reuse or skip.\n\n[1] Sudhakaran+, Gate-Shift Networks for Video Action Recognition, CVPR 2020\n[2] Martinez+, Action recognition with spatial-temporal discriminative filter banks, ICCV 2019\n[3] Jiang+, STM: SpatioTemporal and Motion Encoding for Action RecognitionSTM: SpatioTemporal and Motion Encoding for Action Recognition, ICCV 2019\n[4] Kwon+, MotionSqueeze: Neural Motion Feature Learning for Video Understanding, ECCV 2020", "extra_scores": {}, "extra_text": {}}, {"review_id": "hl8_zqEvBi", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#################################\n\nSummary:\n\nThe paper presented an adaptive inference model for efficient action recognition in videos. The core of the model is the dynamic gating of feature channels that controls the fusion between two frame features, whereby the gating is conditioned on the input video and helps to reduce the computational cost at runtime. The proposed model was evaluated on several video action datasets and compared against a number of existing deep models. The results demonstrated a good efficiency-accuracy trade-off for the proposed model.  \n\n#################################\n\nPros:\n* The paper has a novel idea (adaptive temporal feature fusion) and addresses an important problem in vision (efficient action recognition). \n* Solid experiments on multiple datasets. The analysis of the learned policy is quite interesting.\n* Well-written paper\n\n#################################\n\nCons:\n* Limited technical novelty\n\nThe idea of building adaptive inference models with a policy network for video classification has been previously explored by Wu et al., Meng et al. and others (e.g., skip part of the model, select a subset of frames, choose the input resolution to the model). The main technical component of the model is also very similar to the channel gating network (Hua et al.). The key innovation seems to be the perspective of modeling temporal feature fusion for adaptive inference. This is probably best considered as in parallel to previous approaches for adaptive video recognition. The technical components thus look less exciting.\n\n* Lack of comparison to other adaptive inference models / temporal fusion schemes   \n\nThere isn’t a real comparison between the proposed method and recent works on adaptive inference video recognition (e.g, Wu et al, Meng et al.). The benefit of model temporal feature fusion --- a main contribution of the paper, thus remains unclear with respect to other design choices (e.g., input resolution or frame selection). I’d suggest some experiments that compare to those work. Another important experiment is to contrast the proposed method with other temporal feature fusion schemes (e.g, LSTM, TSM). For example, TSM --- a hand-crafted feature fusion module, seems to have less number of parameters, slightly higher FLOPs and comparable accuracy (Table 3). If that is the case, the contribution of the proposed adaptive fusion scheme is much weakened. \n\n#################################\n\nMinor comments:\n\nIt is not totally clear to me how the FLOPs of the proposed model are computed. As the proposed model will have a different FLOP conditioned on the input video, were the reported FLOPs averaged across the dataset? I was not able to find a description in the paper. \n\nIt will be great if the authors can report some run-time performance (e.g., wall time). To achieve the theoretic FLOPs, the proposed model will rely on filter re-arrangement on the fly and sparse convolution kernels. Both can be less efficient on certain devices, e.g., GPUs.  \n \n\n#################################\n\nJustification for score:\n\nAll in all a good paper. My main concern is the missing link / comparison to previous works on adaptive video recognition. If this concern can be addressed, I am happy to raise my rating. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "H99l-1BAOyN", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors assessed how their adaptive temporal fusion network performs on public datasets such as Something V1&2, Kinetics, etc.. The contribution of this paper is in proposing an approach to automatically determine which channels to keep, reuse, or skip per layer and per target instance that can result in efficient action recognition. \n\nSTRENGTHS: \nThe proposed method is model-agnostic, making it easy to use as a plugin operation for other network architectures. \nReusing history features when necessary to make the network capable for strong temporal modeling. \n\nCONCERNS:\nThe paper has examined the temporal fusion module on BN-Inception and ResNet models, while more recent models’ evaluation is missing. \nWhile the policy network is defined as two FC layers and a ReLU, it is not clear why the authors chose this architecture and how they have tuned it?\nIn section 3, Using 2D-CNN for Action Recognition, a citation to one of the recent works in modeling the temporal causality is missing: Asghari-Esfeden, Sadjad, Mario Sznaier, and Octavia Camps. \"Dynamic Motion Representation for Human Action Recognition.\" In The IEEE Winter Conference on Applications of Computer Vision, pp. 557-566. 2020.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a model for video action recognition.  The reviewers appreciated the development of a novel dynamic fusion method that examines channels from feature maps for use in temporal modeling.  After reading the authors' responses, the reviewers converged on an accept rating.  The solid empirical results and analysis, the fact that is is a plug-in method that could be used in other models, and the clear exposition were deemed to be positives.  As such, this paper is accepted to ICLR 2021.
Decision: Accept (Poster)