VA-RED$^2$: Video Adaptive Redundancy Reduction
Authors: Bowen Pan
Rameswar Panda
Camilo Luciano Fosco
Chung-Ching Lin
Alex J Andonian
Yue Meng
Kate Saenko
Aude Oliva
Rogerio Feris
Venue: ICLR 2021 Poster (2021)

Abstract: Performing inference on deep learning models for videos remains a challenge due to the large amount of computational resources required to achieve robust recognition. An inherent property of real-world videos is the high correlation of information across frames which can translate into redundancy in either temporal or spatial feature maps of the models, or both. The type of redundant features depends on the dynamics and type of events in the video: static videos have more temporal redundancy while videos focusing on objects tend to have more channel redundancy. Here we present a redundancy reduction framework, termed VA-RED$^2$, which is input-dependent. Specifically, our VA-RED$^2$ framework uses an input-dependent policy to decide how many features need to be computed for temporal and channel dimensions. To keep the capacity of the original model, after fully computing the necessary features, we reconstruct the remaining redundant features from those using cheap linear operations. We learn the adaptive policy jointly with the network weights in a differentiable way with a shared-weight mechanism, making it highly efficient. Extensive experiments on multiple video datasets and different visual tasks show that our framework achieves $20\% - 40\%$ reduction in computation (FLOPs) when compared to state-of-the-art methods without any performance loss. Project page: http://people.csail.mit.edu/bpan/va-red/.

Published as a conference paper at ICLR 2021
VA-RED?: VIDEO ADAPTIVE REDUNDANCY REDUC- TION
Bowen Pan', Rameewaar Panda", Camilo Fosco', Chung-Ching Lin", Alex Andonian',
Yue Meng", ; Kate Sacnko*.,, Aude Oliva" 2-2, Rogerio Feris"
MIT CSAIL, 2MIT-IBM Waston AI Lab, Microsoff, *Boston University
ABSTRACT
Performing inference on deep learning models for videos remains challenge due to the large amount of computational resources required to achieve robust recognition. An inherent property of real-world videos is the high correlation of information across frames which can translate into redundancy in either temporal or spatial feature maps of the models, both. The type of redundant features depends on the dynamics and type of events in the video: static videos have more temporal redundancy while videos focusing on objects tend to have more channel redundancy. Here we present a redundancy reduction framework, termed VA-RED", which is input-dependent. Specifically, our VA-RED* framework uses an input-dependent policy to decide how many features need to be computed for temporal and channel dimensions. To keep the capacity of the original model, after fully computing the necessary features, we reconstruct the remaining redundant features from those using cheap linear operations. We learn the adaptive policy jointly with the network weights in differentiable way with shared-weight mechanism, making it highly efficient. Extensive experiments on multiple video datasets and different visual tasks show that framework achieves 20% 40% reduction in computation (FLOPs) when compared to sfate-of-the-ard methods without any performance loss. Project page: httt:://eoppp csai 1. mi t.. edu /bpan/pan///// vvaree//
INTRODUCTION
Large computationally expensive models based on 2D/3D convolutional neural networks (CNNs) are widely used in video understanding (Tran et al. 2015 Carreira & Zisserman, 2017] Tran et al. 201 8). Thus, increasing computational efficiency is highly sought after (Feichtenhoffrr 2020 Zhou et al.
2018c Zolfaghari et al. 2018). However, most of these efficient approaches focus architectural changes in order to maximize network capacity while maintaining a compact model (Zolfagaari et al.
2018 Feichtenhofer 2020) or improving the way that the network consumes temporal information
(Feichtenhofer et al. 2018 Korbar et al.) 2019). Despite promising results, it is well known that CNNs perform unnecessary computations at some levels of the network (Han et al. 2015a Howard et al. 2017 Sandler et al. 2018 Feichtenhofer 20200 Pan et al.) 2018), especially for video models since the high appearance similarity between consecutive frames results large amount of redundancy.
In this paper, we aim at dynamically reducing the internal computations of popular video CNN architectures. Our motivation comes from the existence of highly similar feature maps across both time and channel dimensions in video models. Furthermore, this internal redundancy varies depending on the input: for instance, static videos will have more temporal redundancy whereas videos depicting single large object moving tend to produce a higher number of redundant feature maps. To reduce the varied redundancy across channel and temporal dimensions, we introduce an input-dependent redundancy reduction framework called VA-RED2 (Video Adaptive REDundancy REDuction) for efficient video recognition (see Figure [ for an illustrative example). Our method is model-agnostic and hence be applied to any sfte-of-thhe-r video recognition networks.
The key mechanism that VA-RED2 uses to increase efficiency is to replace full computations of some redundant feature maps with cheap reconstruction operations. Specifically, our framework avoids computing all the feature maps. Instead, we choose to only calculate those non-redundant part of feature maps and reconstruct the rest using cheap linear operations from the non-redundant

---

Published as a conference paper at ICLR 2021
Example#1: adaptive temporal redundancy reduction
Example#2: adaptive channel redundancy reduction
Stem
La
Ly
Output layer
Stem
Ly Output layer
f1
12
"aclut-
mmmle
-ratting"
spaaking
/4
dlashed arrow means closed
nemporal-wise fully-computedd feature
channel-wise toly-computed feature
convolution
charnelf-wise chrap operation
tomporal cheaply gonerated feature
channol gengrratdee feature
temporal cheap aperation
Figure I: Our VA-RED framework dynamically reduces the redundancy in two dimensions. Example 1 (left) shows case where the input video has little movement. The features in the temporal dimension are highly redundant, so our framework fully computes subset of features, and reconstructs the rest with cheap linear operations. In the second example, we show that our framework can reduce computational complexity by performing similar operation over channels: only part of the features along the channel dimension are computed, and cheap operations are used to generate the rest.
features maps. In addition, VA-RED makes decisions on a per-input basis: our framework learns an input-dependent policy that defines "full computation ratio" for each layer of a 2D/3D network. This ratio determines the amount of features that will be fully computed at that layer, versus the features that will be reconstructed from the non-redundant feature maps. Importantly, we apply this strategy on both time and channel dimensions. We show that for both traditional video models such as I3D (Carreira Zisserman 2017). R(2+1)D (Tran et al., 2018), and more advanced models such as X3D (Feichtenhofer, 2020), this method significantly reduces the total floating point operations (FLOPs) on common video datasets without accuracy degradation.
The main contributions of our work includes: (1) A novel for efficient video recognition, VA-RED2, that automatically decides what feature maps to compute per input instance. Our approach is in contrast to most current video processing networks, where feature redundancy across both time and channel dimensions is not directly mitigated. (2) An adaptive policy jointly learned with the network weights in fully differentiable way with shared- weight mechanism, that allows us to make decisions on how many feature maps to compute. Our approach is model-agnostic and can be applied to any backbones reduce feature redundancy in both time and channel domains. (3) Striking results of VA-RED over baselines, with a 30% reduction in computation comparison R(2+1)D (Tran al. 2018 40% over 13D-Inception (Carreiral & Zisserman 2017), and about 20% over the recently proposed X3D-M (Feichtenhofer 2020) without any performance loss, for video action recognition task. The superiority of our approach extensively tested on three video recognition datasets Mini-Kinetice Zisserman 2017). and Moments-In, Time (Monfort et al. 2019)) and one spatio-temporal action localization dataset (J-HMDB-21 (Jhuang et 2013)). (4) A generalization of our framework to video action recognition, spatio-temporal localization, and semantic segmentation tasks, achieving promising results while offering significant reduction in computation over competing methods.
adaptive framework
Kinetics-400 (Carreira
RELATED WORK
Efficiency in Video Understanding Models. Video understanding has made significant progress in recent years, mainly due to the adoption of convolutional neural networks, in form of 2D CNNs (Karpa
thy et al.] 22014] Simonyan & Zisserman, 2014 Chéron et al. 2015 Feichtenhofer et 2017
Gkioxar & Mali]]2015: Wang et al. 2016: Zhou et al. 2018a Lin et al. 22019] 1 Fan et al.
2019
or 3D CNNs (Tran et al. 2015| Carreira Zisserman 2017 Hara et al. 2018] Tran al.
2018).
Despite promising results on common benchmarks, there is a significant interest in developing more
efficient techniques and smaller models with reasonable performance. Previous works have shown reductions in computational complexity by using hybrid 2D-3D architectures (Xie et al.) 2018 Zhou et al., 2018c; Zolfaghari et al. 2018). group convolution (Tran et al.) 2019) or selecting salient clips Korbar et al. 2019). Feichtenhofer et al., (Feichtenhofer et al. 2018) propose a dedicated www-frameraee pathway. Expansion of 2D architectures through a stepwise expansion approach the key variables such temporal duration, frame rate, spatial resolution, network width, is
2

---

Published as a conference paper at ICLR 2021
recently proposed in (Feichtenhofer 2020). Diba et al. (Diba al. videos with a self-supervised task for video understanding. Fan et al. (Fan et al. (2020) incorporate a efficient learnable 3D-shift module into a 3D video network. Wang al. (Wang al., 2020) devise
2019) learn motion dynamic of
correlation module to learn correlation along temporal dimension. Li et al. (Li et al. 2020) encode
the clip-level ordered temporal information with a CIDC network. While these approaches bring considerable efficiency improvements, none of them dynamically calibrates the required feature map computations on a per-input basis. Our framework achieves substantial improvements in average efficiency by avoiding redundant feature map computation depending on the input.
Adaptive Inference. Many adaptive computation methods have been recently proposed with the goal of improving efficiency (Bengio et al., [015 2013) Veit & Belongie 2018) Wang et al. 2018 Graves 2016; Meng et al., 2021). Several works have been proposed that add decision branches to different
layers of CNNs to learn whether to exit the network for faster inference (Yu et al.. 2018 Figurnov
et al. 2017 McGill & Perona 2017 Teerapittayanon et al. 2016) Wang et al. (Wang et al., (018))
propose skip convolutional blocks on a per input basis using reinforcement learning and supervised pre-training. Veit et al. (Veit & Belongie 2018) propose a block skipping method controlled by samples from Gumbel softmax, while Wu et al. (Wu et al. 2018) develop a reinforcement learning approach to achieve this goal. Adaptive computation time for recurrent neural networks is also presented in (Graves) 2016). SpotTune (Guo al.) 2019) learns to adaptively route information through finetuned or pre-trained layers. A few works have been recently proposed for selecting salient frames conditioned on the input (Yeung et al. 2016 Wu et al. 2019 Korbar et al., 2019) Gao et al.. 2019) while recognizing actions in long untrimmed videos. Different from adaptive data sampling (Yeung et al., 2016 Wu et al. 2019; Korbar et al., 2019 Gao et al.) 2019), in this paper, our goal is to remove feature map redundancy by deciding how many features need to be computed for temporal and channel dimensions per input basis, for efficient video recognition. AR-Net (Meng et al. (020) recently learns adaptively choose the resolution of input frames with several individual backbone networks for video inference. In contrast, our method focuses on reducing the redundancy in both temporal and channel dimension, and is applicable to both 3D and 2D models, while AR-Net is only for 2D model and it is focused on spatial resolution. Moreover, our method integrates all the inference routes into a single model which is in the almost same size to the original base model. Thus our model is significantly smaller than AR-Net in terms number of model parameters.
Korbar
Neural Architecture Search. Our network learns the best internal redundancy reduction scheme, which is similar to previous work on automatically searching architectures (Elsken et al., 2018). Liu et al. (Liu al., 2018) formulate the architecture search task in a differentiable manner: Cai et al. (Cai et al. 2018) directly learn architectures for a target task and hardware, Tan et al. (Tan) Le 2019) design a compound scaling strategy that searches through several key dimensions for CNNs (depth, width, resolution). Finally, Tan et al. (Tan et al.) (019) incorporate latency to find efficient networks adapted for mobile use. In contrast, our approach learns a policy that chooses over full or reduced convolutions at inference time, effectively switching between various discovered subnetworks to minimize redundant computations and deliver high accuracy.
3 VIDEO ADAPTIVE REDUNDANCY REDUCTION
Our main goal is to automatically decide which feature maps to compute for each input video in order to classify it correctly with the minimum computation. The intuition behind our proposed method is that there are many similar feature maps along the temporal and channel dimensions. For each video instance, we estimate the ratio of feature maps that need to be fully computed along the temporal dimension and channel dimension. Then, for the other feature maps, we reconstruct them from those pre-computed feature maps using cheap linear operations.
Approach Overview. Without loss of generality, we start from a 3D convolutional network G, and denote its 1th 3D convolution layer as fi, and the corresponding input and output feature maps X and Yi respectively. For each 3D convolution layer, we use a very lightweight policy layer pt denoted as soft modulation gate decide the ratio of feature maps along the temporal and channel dimensions which need to be computed. As shown in Figure 2 for temporal-wise dynamic inference, we reduce the computation of 3D convolution layer by dynamically scaling the temporal stride of the 3D filter with factor R- 2Pt (X)] Thus the shape of output Y/' becomes Cout x To/ R H, W.. To keep the same output shape, we reconstruct the remaining features based on Y/ as
Vili + iR] = {
if E {1...., R -11 1} , (0. 1, ...., To/R. R 1}, x i
Φ (Y/(I)
(1)
3

---

Published as a conference paper at ICLR 2021
cuxiii
cxxxxx
avg. pool
G.TxXXX
C
Sputial Conv.
5:1
(ptidd:
Temporal polcy:
Channel polay:
Idendiy
Output features
D
Concat
(a) Temporal-wise dynamic convolution
(b) Channel-wise dynamic convolution
Figure 2: An illustration of dynamic convolution along temporal dimension (a) and channel dimension (b) respectively. &, and Φ represent the temporal cheap operation and spatial cheap operation respectively. In (a), we multiply the temporal stride S with the factor R = 2 to reduce computation, where Pt is the temporal policy output by soft modulation gate. In (b), we compute part of output features with the ratio of (}))r, where Pe is the channel policy. Best viewed in color.
where Yilj iR] represents the (j + iR)* feature map Yi along the temporal dimension, Y/[i] denotes the jth feature map of Y/, and $ is the cheap linear operation along the temporal dimension. The total computational cost of this process can be written as:
c( 1 c(fi) + Ec(s' ~ R
1 R
c(f)).
(2)
where the function C(.) returns the computation cost for specific operation, and f represents our dynamic convolution process along temporal dimension. Diffcrent from temporal-wise dynamic inference, we reduce the channel-wise computation by dynamically controlling the number of output channels. We scale the output channel number with factor r (2 (x))" In this case, the shape of output Y/ rCout x To x H, W,, Same as before, we reconstruct the remaining features via cheap linear operations, which can be formulated as Y; [Y (Y(')) where (T(Y)) E R(I-r)Couu HI, xW. represents the cheaply generated feature maps along the channel dimension, and Yi E RCoax XT, x..x W. is the output of the channel-wise dynamic inference. The total computation cost of joint temporal tansera--wise wise and channel-wize -wise dynamic inference is:
c(f
R C(fi).
(3)
where f: is the adjunct process of temporal-wisc and channel-wise dynamic inference.
Soft Modulation Gate for Differentiable Optimization. We adopt an extremely lightweight policy layer pl called soft modulation gate for each convolution layer fi to modulate the ratio of features which need to be computed. Specifically, the soft modulation gate takes the input feature maps X, as input and learns two probability vectors V R and V RSa, where S₁ and S, the temporal search space size and the channel search space size respectively. The V and Vt are learned by:
[VL V (X() ((W G(X))) 8%)).
(4)
where F(:,.) denotes the cullecco layer, A' is the batch normalization, 5(-) represents the tanh(-) function, G is the global pooling operation whose output shape is C.n. T x 1 x 1, o(.) is the output activation function, here just use max(tanh(.), 0) whose output range is [0, 1), and the weights of their corresponding layers, D,, is the hidden dimension number. V and V will then be used to modulate the ratio of the feature maps to be computed in temporal-wist dynamic convolution and channel wise dynamic convolution. During training, obtain the final output of the dynamic convolution by weighted sum of all the feature
Wp.1 E x( Wp.2 DD, xC..
4

---

Published as a conference paper at ICLR 2021
maps which contains different ratio of fully-computed features as follows:
S, Y - 2- Yi Σ V R
(5)
where [8(:;) r) is the channel-wise dynamic convolution with the channel scaling factor r, and ff R) it the temporal-wise dynamic convolution with the temporal stride scaling factor R. During the inference phase, only the dynamic convolutions whose weights are not zero will be computed.
Shighd-weight Training and Inference. Many works in adaptive computation and neural architec- ture search suffer from very heavy computational cost and memory usage during training stage due to the large search space. In our case, under the naive implementation, the training computational cost and parameter size would linearly grow as the search space size increases. To train our model efficiently, we utilize a weight-sharing mechanism to reduce the computational cost and training memory. To be specific, we first compute all the possible necessary features using a big kernel. Then, for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. Though this, we are able to keep the computational cost at constant value invariant to the search space. More details on this are included in Section B of the Appendix.
Efficiency Loss. To encourage our network to output computational efficient subgraph, we introduce the efficiency loss La during the training process, which can be formulated as
L = Σ 1=1 C())
C(U)
7i T pm = { R
1 if correct otherwise
(6)
where rf" is channel scaling factor of the largest filter in the series of channel- wise dynamic convo- lutions, and Re is stride scaling factor of the largest filter of temporal- dynamic convolutions. Overall, the loss function of our whole framework can be written as L La + Xahes where I. is the accuracy loss of the whole network and Xe is the weight of efficiency loss which can be used to balance the importance of the optimization of prediction accuracy and computational cost.
4 EXPERIMENTS
Datasets. We conduct our video action recognition experiments on three standard benchmarks: Mini-Kinetics-222 2000, Kinetics-400, and Moments-In- Time. Mini-Kinetics-2D9 (assembled by (Meng et al., 2020)) is subset of full Kinetics dataset (Carreira & Zisserman, 2017) containing 12Tk videos for training and 10k videos for testing across 200 action classes. Moments-In-
dataset has 802,244 videos in training and 33,900 videos in validation across 339 categories. To show the generalization ability to different task, we also conduct the video spatio-tempors action localiza- tion on J H-HMDB22 21 (Jhuang et al. 2013). J-HMDB-21 21 is subset of HMDB dataset (Kuehne et al.) 2011} which has 928 short videos with 21 action categories. We report results on the first split. For semantic segmentation experiments, we use ADE20K dataset (Zhou al. 2017 2018b), containing 20k images for training and 2k images for validation. ADE20K is a densely labeled image dataset where objects and object parts are segmented down to pixel level. We report results on validation set.
Model Architectures. We evaluate our method on three most widely-used model architectures: I3D Carrreira & Zisserman| 2017), R(2+1)D (Tran et al., 2018), and the recent efficient model X3D (Feichtenhofer 2020). We consider 13D- BBD-Inception V2 (denoted as I3D below) and R(2+1)D-18 18 (denoted as R(2+1)D below) as our base model. In our implementation of X3D, we remove all the swish non-linearity (Ramachandran et al., 2017) except those in SE layer (Hu et al.) 2018) to save training memory and speed up the inference speed on GPU. We choose X3D-M (denote as X3D below) our base model and demonstrate that our method is generally effective across datasets.
Implementation Details. We train and evaluate our baseline models by mainly following the settings in their original papers (Tran et al. 2018 Xie et al. 2018 Feichtenhofer 2020). We train all our base and dynamic models for 120 epochs on mini-Kinetics Kinetics- Kinetics-4uP 400, and 60 epochs on Moments- -n-Time dataset. We use mini-batch size of 12 clips per GPU and adopt synchronized SGD with cosine learning rate decaying strategy (Loshchilov & Hutter] 2016) to train all our models. Dynamic models are finetuned with efficiency loss for 40/20 epochs to reduce density of inference
5

---

Published as a conference paper at ICLR 2021
Table I: Action recognition results using different number of input frames and difffrent search space. We choose R(2+1)D-188 on Mini-Kinetics-200 and study the performance with different number of input frames and different search space (denoted as Sea. Sp.). Search space of means that both temporal-wise and channel-wis policy network have 2 alternatives: computing all feature maps, or computing only 2)) of the feature maps. Similarly, search space 3 have 3 alternatives: computing I) all feature maps, 2) 2 of feature maps, 3) 1 of feature maps. X denote the base model and / denote the dynamic model trained using our proposed approach VA-RED?. We also report the average speed of different models in terms of number of clips processed in one second seciodd)
length
sp.
GFLOPs GFL
GFLOPs
avg speed
clip-1 1
video-1
vidco-5
x
27.7
27.7
27.7
192.1
56.4
66,8
86.8
8
2
20.0(-28%)
22.1(-209%
18.0(-35%)
205.5
57.7
68.0
87.4
3
21.6(-22%%
23.2(-16%)
19.8(-29%
201.4
58.2
67.7
87.4
X 55.2
55.2
55.2
97.1
57.5
67.5
87.1
16
2
40.4(-27%%
43.2(-22%
36 36.6(-34%)
108.7
60.6
70.0
88.7
X
110.5
110.5
110.5
19.6
60.5
69.4
88.2
32
2
79.3(-28%)
89.5(-19%)
53.4
63.3
72.3
89.7
Table 2: Action recognition results Mini- Kinetics-200. We set the search space as 2 and train all the models with 16 frames. The metric speed uses c/pp second as the unit.
Table 3: Action recognition results with Temporal Pyramid Network (TPN) on Mini- Kinetics-200. TPN-8f and TPN-16f indicate that we use 8 frames and 16 frames as input to the model respectively.
Model
Dy.
GFLOPs
Speed
clip-1
video-1
R(2+1)D
X
55.2
97.1
57.5
67.5
Model
Dy.
GFL.PPs
clip-1
vidco-1
V
40.4
108.7
60.6
70.0
x 28.5
58.9
67.2
I3D
56.0
116.4
59.7
68.3
TPN-8f
/
21.5
59.2
68.8
26.5
141.7
62.2
71.1
X3D
6.20
5.03
178.2
169.4
65.5
66.5
72.1
72.2
TPN-16f
X
56.8
41.5
59.8
60.8
68.5
70.6
graph while maintaining the accuracy. During finetuning, we set X to 0.8 and learning rate to 0.01 for R(2+1)D and 0.1 for 13D and X3D. For testing, we adopt K-LefiCenterRiihh strategy: K temporal clips are uniformly sampled from the whole video, on which we sample the left, center and right crops along the longer spatial axis, the final prediction is obtained by averaging these 3 x K clip predictions. We set ' 10 on Mini-Kinetics-D 200 and Kinetics-400 and K = 3 on Moments- In-tts-II-iiinn Time. More implementation details are included in Section of Appendix. For video spatio-temporal action localization, we adopt YOWO architecture in (Köpūklü et al.] (019) and replace 2D branch with 3D backbone to directly compare them. We freeze the parameters of 3D backbone as suggested in (Köpūkūū et al. 2019) due to small number of training video in J-HMDB-21 (Thuang et al., 2013). The rest part of the network is optimized by SGD with initial learning rate of 10 - Learming rate reduced with a decaying factor of 0.5 at 10k, 20k, 30k and 40k iterations. For semantic segmentation, we conduct experiments using PSPNet (Zhao et al. 2017). with dilated ResNet- (Yu al. 2017 He et al. 2016) as our backbone architecture. As PSPNet is devised for image semantic segmentation, we only apply the channel-wise redundancy reduction the model and adopt synchronized SGD training for 100k iterations across 4GPUs with 2 images on each GPU. The learning rate decay follows the cosine learning rate schedule schedule (Loshchilov & Hutter 2016).
Results on Video Action Recognition. We first evaluate our method by applying it to R(2+1)D- 18 (Tran et al. 2018) with different number of input frames and different size of search space. Here we use GFLOPs (floating point operations) to measure the computational cost of the model and report clip-1, video-1 and video-5 metrics to measure the accuracy of our models, where clip-1 is the top-1 accuracy of model evaluation with only one clip sampled from video, video-1 and video-5 are the top-1 and top-5 accuracy of model evaluated with K-LefiCenterRight strategy. Note that we report the FLOPs of single video clips at the spatial resolution 256 x 256 (for I3D and X3D) or 128 x 128 (for R(2+1)D). In addition, we report the speed of each model with the metric of clip/ second, which denotes the number of video clips that are processed in one second. We create the environment with Torch 1.6, CUDA 11.0, and a single NVIDIA TITAN RTX (24GB) GPU as our testbed to measure speed of different models. Table T shows the results (In all of the tables, represents the
6

---

Published as a conference paper at ICLR 2021
Table 4: Comparison with CorrNet (Wang et al. 2020) and AR-Net (Meng al., 2020) on Mini-Kinetics-200. We set the search space as 2 and train all the models with 16 frames.
Model
Dy.
GFLOPS
clip-1
vidco-1
Method
Params
GFLOPs
clip-1
x
60.8
59.9
68.2
AR-Net
63.0M
44.8
67.2
CorrNet
45.5
60.4
70.0
VA-RED"
23,9M
43.4
68.3
Table 5: Action recognition results on Kinetics-400. We set the search space as 2, meaning models can choose to compute all feature maps or 2 of them both on temporal and channel-wise convolutions.
16-frume
32-frame
Model
Dy.
GFLOPs
speed
clip-1
vidco-1
vidco-5
GFLOPs
speed
clip-1
vidco-1
vidco-5
X
55.2
97.1
57.3
65.6
86.3
110.5
49.6
61.5
69.0
85.6
R(2+1)D
40.3
105.9
58.4
67.6
87.6
80.7
53.0
61.5
70.0
88,9
X
56.0
116.4
55.1
66.5
86.7
112.0
57.6
57.2
64.9
86.5
I3D
/
32.1
140.7
58.6
67.1
87.2
64.3
71.7
61.0
68.6
88.4
X3D
X
6.42
169.4
63.2
70.6
90.0
[X3D-M is designed for 16 frames]
X 5.38
177.6
65.3
72.4
90.7
original fixed model architecture while / denote the dynamic model trained using our proposed approach). Our proposed approach VA-RED² significantly reduces the computational cost while improving the accuracy. We observe that dynamic model with the search space size of 2 has the best performance in terms of accuracy, GFLOPS and speed. We further test our VA-RED with all of the three model architectures: the very recent temporal pyramid module (Yang et al. 2020) and correlation module (Wang et al. 2020) on Mini-Kinne Kinetics-200 dataset. We choose R(2+1)D-18 18 with TPN and CornNet the backbone architecture and test the performance of our method using a search space of in Table 3 and Table 4 (Left) respectively. Table Tsoos that method boosts the speed of base I3D-Inception V2 and R(2+1)D models by 21.7% and 10.6% respectively, showing its advantages not only in terms of GFLOPS but also in actual speed. Table 4 (Left) shows that our dynamic approach also outperforms the baseline CornNet by 1.8% in top-1 video accuracy, while reducing the computational cost by 25.2% on Mini- Simii-Knntiiss200. Furthermore, we compare our method with AR-Net (Meng al.) 2020), which is a recent adaptive method that selects optimal input resolutions for vidco inference. We conduct our experiments on 16-frame TSN (Wang et al. 2016) with ResNet50 backbone and provide the comparison on FLOPs, parameter size, and accuracy (Table[[ (Right). To make a fair comparison, we train AR-Net using the official implementation on the same Mini-Kinetice initialization (He et al. 2015)) T (Right) shows that our method, VA-RED outperforms AR-Net and Table show
1)D-18, I3D- 13D-Inneption V2, and X3D-M (Table 2 including
dataset with Kaiming
both accuracy and GILOPS, while using about 62% less parameters. Table 5 the results of different methods on Kinetics 400 and Moments In- Time, respectively. To summarize, we observe that VA-RED2 consistently improves the performance of all the base models including the recent architectures X3D, TPN, and CorNNet, while offering significant reduction in computation. Moreover, our approach is model-agnostic, which allows this to be served as a plugin operation for a wide range of action recognition architectures. From the comparison among different models, we find that our proposed VA-RED* achieves the most computation reduction on BBD-Inccption V2, between 40% and 50%, while reducing less than 20% on X3D-M. This because X3D-M is already very efficient both in terms of channel dimension and temporal dimension. Notice that the frames input X3D-M are at the temporal stride of which makes them share less similarity. Furthermore, we observe that dynamic Inne-Incn V2 has very little variation of the computation for different input instances. This could be because of the topology configuration of the Inception V2, which has lots of parallel structures inside the network architecture
We also compare VA-RED with a weight- level pruning method (Han et al. 2015b) and a automatic channel pruning method (CGNet) (Hua et al. (209)) approach significantly outperforms the weight-level pruning method by a margin of about 3%-4% in clip-1 accuracy with similar computation over the original fixed model and consistently outperforms CGNet while requiring less GFLOPs (maximum 2.8% in 16 frame). These results well demonstrate the effectiveness of our dynamic video redundancy framework over network pruning methods.
Table 7 shows that our
7

---

Published as a conference paper at ICLR 2021
Table 6: Action recognition results on Moments- In-Time. We set the search space as 2, i.c. mod- els can choose to compute all feature maps them both temporal and channel-wist convolu- tions. The speed uses clip/ second as the unit.
Table 7: Comparison with network pruning methods. We choose R(2+1)D on Mini-Kineetics- 200 dataset with different number of input frames. Numbers in green/blue quantitatively show how much our proposed method is better/worse than these pruning methods.
of
Model
Dy.
GFLOPs
speed
clip-1
vidco-1
Method
Frames
GFLOPs
clip-1
X
55.2
97.1
27.0
28.8
8
19.9 (-0.1)
54.5 (+3.2)
R(++1)D
/
42.5
105.5
27.3
30.1
Weight-level
16
40.3 (-0.1)
57.7 (+2.9)
32
79.6 (-0.3)
59.6 (43.7)
56.0
116.4
25.7
26.8
13D
32.1
140.7
26.3
28.5
8
23.8 (+3.8)
56.2 (+1.5)
x
6.20
169.4
24.8
24.8
CGNet
16
47.6 (+7.2)
57.8 (+2.8)
X3D
5.21
177.4
26.7
27.7
32
95.3 (+16.0)
61.8 (+1.5)
Table 8: Action localization results J-HMDB. We set the search space 2 for dynamic models. The speed uses clip/ second as the unit.
Table 9: Effect of efficiency loss on Kinetics-
400. Eff. denotes the efficiency loss.
Model
Eli.
GFLOPs
clip-1
video-1
Model
Dy.
GFLOPs
speed
mAP
Recall
Classif.
No
49.8
57.9
66.7
X
43.9
141.1
44.8
67.3
87.2
R(2+1)D
Yes
40.3
58.4
67.6
/
21.3
167.4
47.2
05.6
91.1
X3D
/
x
4.85
5.75
176.3
184.6
47.9
50.0
65.2
65.8
93.0
93,2
13D
Yes
No
32.1
56.0
58.0
58.6
66.5
67.1
Table 10: Ablation experiments on dynamic modeling along temporal and channel dimensions. We choose R(2+1)D-18 Mini-Kinetics-P01 and set the search space to in all the dynamic models.
8-frame
16-frame
Dy. Temp.
Dy. Chan.
GFLOPs
speed
clip-1
video-1
GFLOPs
speed
clip-1
video-1 1
X
27.7
192.1
56.4
66.8
55.2
97.1
57.5
67.5
X
23.5
198.6
57.1
66.8
46.1
105.0
58.6
67.6
22.7
196.5
57.0
66.7
46.3
102.0
59.2
68.3
20.0
205.5
57.7
68.0
40.4
108.7
60.6
70.0
Action Localization. We further extend our method to the spatio- temporal action localization task to demonstrate the generalization ability to different task. We conduct our method on J-HMDB-2 21 with two different 3D backbone networks: 13D-Inception and X3D-M. We report frame-mAP at IOU threshold 0.5, recall value at IOU threshold 0.5, and classification accuracy of correctly localized detections to measure the performance of the detector. Table [8] shows that our dynamic approach outperforms the baselines on all three metrics while offering significant savings in FLOPs (e.g., more than 50% savings I3D). In summary, VA-RED is clearly better than the baseline architectures in terms of both accuracy and computation cost on both recognition and localization tasks, making it suitable for efficient video understanding.
Results on
Effect of Efficiency Loss. We conduct an experiment by comparing the model performance before and after being finetuned with our proposed efficiency loss. shows that finetuning our dynamic model with efficiency loss significantly reduces the computation without any accuracy loss.
Ablation Experiments on Dynamic Modeling. We test performance of approach by turning of dynamic modeling along temporal and channel dimensions on Mini-Kinetics-20 Table IO shows that dynamic modeling along both dimensions obtains the best performance while requiring the least computation. This shows importance of input-dependent policy for deciding how many features need to be computed for both temporal and channel dimensions.
Visualization and Analysis. To better understand the policy decision process, we dissect the network layers and count the ratio of feature maps that are being computed during each convolution layers for each category. From Figure 3 we observe that: In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wist features at the early stage and compute stage of the architecture. The channel-wiss policy has also more variation than the temporal- policy among different categories. Furthermore, show few contrasting examples which are in the
channel-wis wiie features at the late
8

---

Published as a conference paper at ICLR 2021
W Channee-- wite pollyy of X3D-M Myers for each category
Tempporsiwwon policy of XSD 1DD-M layers for each cotegory
agalyi
CAR Gaciigg
layers
wise layers
Figure 3: Ratio computed feature per layer and on Mini-Kinetics-225 dataset. We pick the first 25 classes of Mini-Kinn Kinetics-222 200 and visualize the per-block policy X3D-M on each class. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed.
More computation: 66.7%
More computation: 66.2% 2%
More computation: 65.8%
Less compulstion: 52.9%
Less computation: 53.8%
Less computation: 52.1%
cooking eggs
playing volleyball
flipping pancake
Figure 4: Validation video clips from Mini--inntics-2000 For each category. we plot two input video clips which consume the most and the least computational cost respectively. We infer these video clips with 8-frame dynamic R(2+1)D-18 model trained Mini- Mini-Kinetics-200 and the percentage indicates the ratio of actual computational cost 2D convolution to that of the original fixed model. Best viewed color.
Table 11: VA-RED on semantic segmentation. We choose dilated ResNet- 18 as our backbone architecture and set the search space Models are trained for 100K iterations with batch size of 8.
Original model
Channel-wise reduction using VA-RED2
Model
GFLOPs
IoU
GFLOPsaeg
GFLOPsmar
GFLOPSmin
IoU
Dilated ResNet-18
10.6
31.2%
7.8
9.1
7.3
31.3%
same category while requiring very different computation in Figure 4 Video clips which have more complicated scene configuration (e.g. cooking eggs and playing volleyball) and more violent camera motion (e.g. flipping pancake) tend to need more feature maps to do the correct predictions. More qualitative results can found Section Section
F and Section G of the Appendix.
2 on Dense Visual Tasks. Our VA-RED2 framework is also applicable for some dense visual tasks, like semantic segmentation, which requires the pixel-level prediction for the input content. To prove this, we apply our method to semantic segmentation model the ADE-20K dataset (Zhou et al. 2017 2018b). We report computational cost of model encoder and the mean IoU (Intersection Over- Union) in Table As be seen from Table I1 proposed VA-RED? has the absolute advantage in terms of efficiency while maintaining the precision segmentation. This experiment clearly shows that method is only effective the recognition and detection tasks, but also applicable to the dense visual tasks like semantic segmentation.
5 CONCLUSION
In this paper, we propose an input-dependen adaptive framework called VA-RED² for efficient inference which can be easily plugged into most existing video understanding models signifi- cantly reduce the model computation while maintaining the accuracy. Extensive experimental results on video action recognition, spatio-temporal localization, and semantic segmentation validate the effectiveness of our framework in multiple standard benchmark datasets.

---

Published as a conference paper at ICLR 2021
Acknowledgements. This work is supported by IARPA via DOI/IBC contract number D17PC00341. This work is also supported by the MIT-IBM Watson AI Lab.
Disclaimer. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies or endorsements, either expressed or implied, IARPA, DOI/IBC, or the U.S. Government.
REFERENCES
Emmanuel Bengio, Pierre-Luc Bacon, Joelle Pineau, and Doina Precup. Conditional computation in neural networks for faster models. arXiv preprint arXiv:/511.06297 2015.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint
2013.
Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search on target task and hardware. arXiv preprint arXiv:1812.0000 00332, 2018.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Guilhem Chéron, Ivan Laptev, and Cordelia Schmid. P-enn: Pose-based enn features for action recognition. In Proceedings of the IEEE international conference on computer vision, pp. 3218- 3226, 2015.
Ali Diba, Vivek Sharma, Luc Van Gool, and Rainer Stiefelhagen. Dynamonet: Dynamic action and motion network. In Proceedings of the IEEE International Conference Computer Vision, pp. 6192-6201, 2019.
Thomas Elsken, Jan Hendrik Metzen, and Frank Hutter. Efficient multi-objective neural architecture search via lamarckian evolution. arXiv preprint arXiv:1804.0900 09081, 2018.
Linxi Fan, Shyamal Buch, Guanzhi Wang, Ryan Cao, Yuke Zhu, Juan Carlos Niebles, and Li Fei-Fei. Rubiksnet: Learnable 3d-shift for efficient video action recognition. In European Conference on Computer Vision, pp. 505-521. Springer, 2020.
Quanfu Fan, Chun-Fu Richard Chen, Hilde Kuehne, Marco Pistoia, and David Cox. More is less: Learning efficient video representations by big-little network and depthwise temporal aggregation. In Advances Neural Information Processing Systems, pp. 2261-2270, 2019.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. arXiv preprint
arXiv:2004.04730, 2020.
Christoph Feichtenhofer, Axel Pinz, and Richard P Wildes. Spatiotemporal multiplier networks for video action recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4768-4777, 2017.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition, 2018
Michael Figurnov, Maxwell Collins, Yukun Zhu, Li Zhang, Jonathan Huang, Dmitry Vetrov, and Ruslan Salakhutdinov, Spatially adaptive computation time for residual networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1039-1048, 2017.
Ruohan Gao, Tae-Hyun Oh, Kristen Grauman, and Lorenzo Torresani. Listen to look: Action recognition by previewing audio. arXiv preprint arXiv: 1912.4491 2019.
Georgia Gkioxari and Jitendra Malik. Finding action tubes. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 759-768, 2015.
Alex Graves. Adaptive computation time for recurrent neural networks. arXiv: 1eo:1600300.8. 2016.
arXiv preprint
10

---

Published as a conference paper at ICLR 2021
Yunhui Guo, Honghui Shi, Abhishek Kumar, Kristen Grauman, Tajana Rosing, and Rogerio Feris. Spottune: transfer learning through adaptive fine-tuning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4805-4814 2019.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprini
2015a. 5a.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. In Advances in neural information processing systems, pp. 2015b.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6546-6555, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference computer vision and pallern recognition, pp. 770-778, 2016.
Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 2017.
Jie Hu, Li Shen, and Gang Sun. gqueeze-and-excitalee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Weizhe Hua, Yuan Zhou, Christopher M De Sa, Zhiru Zhang, and G Edward Suh. Channel gating neural networks. Advances in Neural Information Processing Systems, pp. 1886-1896, 2019.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint 1rXiv:1502.03167, 2015.
Hueihan Jhuang, Juergen Gall, Silvia Zuffi, Cordelia Schmid, and Michael Black. Towards understanding action recognition. In Proceedings the IEEE international conference computer vision, pp. 3 3192-3199, 2013.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung. Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 1725-1733 1732, 2014.
Okan Köpüklü, Xiangyu Wei, and Gerhard Rigoll. You only watch once: A unified enn architecture for real-time spatiotemporal action localization. 2019.
Bruno Korbar, Du Tran, and Lorenzo Torresani. Sesampler: Sampling salient clips from video for efficient action recognition. In Proceedings of the IEEE International Conference on Computer Vision, pp. 6232-6242, 2019.
Hildegard Kuchne, Hucihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In 201 International Conference Computer Vision, pp. 2556-2561 IEEE, 2011.
Xinyu Li, Bing Shuai, and Joseph Tighe. Directional temporal modeling for action recognition. In European Conference on Computer Vision, pp. 275-291. Springer, 2020.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient vidco understanding. In Proceedings of the IEEE International Conference on Computer Vision, pp. 7083-7093, 2019.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint arXiv: 1806 090550 2018.
11

---

Published as a conference paper at ICLR 2021
Iyya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 03983, 2016.
Mason McGill and Pietro Perona. Deciding how to decide: Dynamic routing in artificial neural networks. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 2363-2372, 2017.
Yue Meng, Chung- mhigg-hhng Lin, Rameswar Panda, Prasanna Sattigeri, Leonid Karlinsky, Aude Oliva, Kate Saenko, and Rogerio Feris. Ar-net: Adaptive frame resolution for efficient action recognition. 2020.
Yue Meng, Rameswar Panda, Chung-Ching Lin, Prasanna Sattigeri, Leonid Karlinsky, Kate Saenko, Aude Oliva, and Rogerio Feris. Adafuse: Adaptive temporal fusion network for efficient action recognition. In International Conference on Learning Representations, 2021.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, al. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine intelligence, 42(2):502-508, 2019.
Bowen Pan, Wuwei Lin, Xiaolin Fang, Chaoqin Huang, Bolei Zhou, and Cewu Lu, Recurrent residual module for fast inference in videos. In The IEEE Conference on Computer Vision and Putlern Recognition (CVPR), June 2018.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint 1rXiv:1710.05941, 2017.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems, pp. 568-576, 2014.
Mingxing Tan and Quoc V Le. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint urxiv:/905 1946, 2019.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Surat Tecrapittayanon, Bradley McDanel, and Hsiang- Kung. Branchynet: Fast inference via early exiting from deep neural networks. In 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 2464-2469. IEEE, 2016.
Du Tran, Lubomir Bourdev, Rob Fergus. Lorenzo Torresani. and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489-4497, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look spatiotemporal convolutions for action recognition. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 6450-6459, 2018.
Du Tran, Heng Wang, Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5552-5561, 2019.
Andreas Veit and Serge Belongie. Convolutional networks with adaptive inference graphs. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 3-18, 2018.
Heng Wang, Du Tran, Lorenzo Torresani, and Matt Feiszli. Video modeling with correlation networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 352-361, 2020.
12

---

Published as a conference paper at ICLR 2021
Limin Wang, Yuanjun Xiong, Zhe Wang. Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pp. 20-36. Springer, 2016.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: Learning dynamic routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV). pp. 409-424, 2018.
Zuxuan Wu, Tushar Nagarajan, Abhishek Kumar, Steven Rennie, Larry S Davis, Kristen Grauman, and Rogerio Feris. Blockdrop: Dynamic inference paths in residual networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 8817-8826, 2018.
Zuxuan Wu, Caiming Xiong, Chih- Yao Ma, Richard Socher, and Larry S Davis. Adaframe: Adaptive frame selection for fast video recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1278-1287, 2019.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 305-3033000 005-321,222 2018.
Ceyuan Yang. Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. End-to-end learning of action detection from frame glimpses in vidcos. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2678-2687, 2016.
Fisher Yu, Vladlen Koltun, and Thomas Funkhouser. Dilated residual networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 472-480, 2017.
Jiahui Yu, Linjie Yang, Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint 1rXiv:1812.0880 2018.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 803-818, 2018a.
Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal on Computer Vision, 2018b.
Yizhou Zhou, Xiaoyan Sun, Jue Zha, and Wenjun Zeng. Mict: Mixed 3d/2d convolutional tube for human action recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 458, 2018c.
Mohammadreza Zolfaghari, Kamaljeet Singh, and Thomas Brox. Eco: Efficient convolutional network for online video understanding, In Proceedings of the European Conference on Computer Vision (ECCV), pp. 695-712, 2018
13

---

Published as a conference paper at ICLR 2021
A DATASET DETAILS
We evaluate the performance of our approach using three video action recognition datasets, namely Kini-Kinetics-2000 (Meng et al. 2020). Kinetics-400 (Carreira & Zisserman 2017), and Moments- In- In-Time (Monfort et al. 2019) and one spatio-temporal action localization task namely J-HMDB- large dataset containing 400 action classes and 240K training videos that are collected from YouTube. The Mini-Kinetie dataset contains 121K videos for training and 10K videos for testing, with each video lasting 6-10 seconds. The original Kinetics dataset is publicly available to download at at deepmi ind. nd. com/ resea rch/ oopnn-pen---e sourco re kinetics We use the official trininn/vaalddatiooneesinn splits of Kinetics-400 and the splits released by authors in (Meng et al. 2020) for Mini-Kinetics-200 in our experiments.
21 (hhuang et al. 2013). Kinetics-400
Moments-in-time (Monfort et al. 2019) is a recent collection one million labeled videos, involving actions from people, animals, objects or natural phenomena. It has 339 classes and each video clip is trimmed to seconds long. This dataset is designed to have a very large set of both inter-class and intra-class variation that captures dynamic event at different levels of abstraction (i.e. "opening" doors, curtains, mouths, even a flower opening its petals). We use the official splits in our experiments. The dataset is publicly available to download at ftttt:://///. chttp:://momeee /wtp/ntommmmt.m///mm//oo csai mi miit.. mer.teedii edu/
Joints for the HMDB dataset (J-HMDB-21 (Jhuang et al. 2013) is based on 928 clips from HMDB51 comprising 21 action categories. Each frame has 2D pose annotation based on a 2D articulated human puppet model that provides scale, pose, segmentation, coarse viewpoint, and dense optical flow for the humans in action. The 21 categories kick ball, pick, pour, pull-up. push, run, shoot ball, shoot bow, shoot gun, sit, stand, swing baseball, throw, walk, wave. The dataset is available download at phttt::// //jj:////mmbb is tue mpg..p...... de/
brush hair, catch, clap, climb stairs, golf, jump,
B IMPLEMENTATION DETAILS
Details of weight Training and Inference. In this section, we provide more details of the shared-weight mechanism presented in Section 3 of the main paper. We first compute all the possible necessary features using a big kernel and then for each dynamic convolution with different scaling factor, we sample its corresponding ratio of necessary features and reconstruct the rest features by cheap operations to get the final output. For example, the original channel-wise dynamic convolution ratio r = ((-() 1) can be analogized to
[(f 1 (5
(49
(5) ( (Cout))) (7)
where : is the index operation along the channel dimension, and ig is the index of the largest channel-wise filter, during training phase, we have ig 1, while during inference phase, i is the smallest index for V2 s.t. V.I.V! [r] = O. By utilizing such a weight mechanism, the computation of the total channel-wise dynamic convolution is reduce to ( C(fi). Further, we have the total computational cost of the adjunct process as
c( (G)tt- .((i),
where i is the index of largest temporal-wise filter.
Training and Inference. We apply our method mainly 2D convolutions in R(2+1)D since 2D convolution takes the most computational cost compared with ID convolution. We train most of our models on 96 NVIDIA Tesla V100-32GB GPUs and perform synchronized BN (Ioffe & Szegedy 2015) across all the GPUs. For R(2+1)D (Tran et al., 2018), the learning rate is initialized as 0.18 and the weight decay is set to be 10-1. For I3D (Carreira & Zisserman, 2017 Xie et al. 2018) and X3D (Feichtenhofer, 2020), the learning rates both start from 1.8 and weight decay factors are x 10 and 5 x 10 respectively. Cosine learning rate decaying strategy is applied to decrease the total learning rate. All of the models are trained from scratch and warmed up for 15 epochs on memi--ineticsKinneeeee
8 epochs on Moments-I In- -n-imme dataset. We adopt the Nesterov momentum optimizer with an initial weight of 0.01 and momentum of 0.9. During training, follow the data augmentation (location jittering, horizontal flipping. corner cropping, and scale jittering) used in TSN (Wang al. 2016) augment the video with different sizes spatially and flip
14

---

Published as a conference paper at ICLR 2021
Table 12: Quantitative results of redundancy experiments. We compute the correlation coefficient, RMSE and redundancy proportions (RP) for feature maps in well-known pretrained video models on Moments-in coments-in-Time -ime and Kinetics-400 datasets. RP is calculated as the number of tensors with both CC and RMSE above redundancy thresholds of 0.85 and 0.001, respectively. We show results corresponding to averaging the per layer values for all videos in the validation sets. We observe that networks trained on Homents-In-Tine In-mime (and evaluated on the Moments in Time validation set) tend to present slightly less redundancy than their Kinetics counterparts, and the time dimension tends to be more redundant than the channel dimension in all cases. We observe severe redundancy across the board (with some dataset-model pairs achieving upwards of 0.8 correlation coefficient between their feature maps), which further motivates redundancy reduction approach.
Datasct
Model
Dimension
CC
RMSE
RP
I3D
Temporal
0.77
0.083
0.62
I3D
Channel
0.71
0.112
0,48
Moments-In- In-e
R(2+1)D
Temporal
0.73
0.108
0.49
R(++IDD
Channel
0.68
0.122
0.43
B3D
Temporal
0.81
0.074
0.68
I3D
Channel
0.76
0.091
0.61
Kinetics-400
R(2+1)D
Temporal
0.78
0.081
0.64
R(2+1)D
Channel
0.73
0.088
0.58
the video horizontally with 50% probability. We use single-clip, center-crop FL.OPs basic unit of computational cost. Inference-timg computational cost is roughly proportional to this, if a fixed number of clips and crops is used, is for our all models. Note that Kinetics-400 dataset is shrinking in size (~155% videos removed from original Kinetics) and the original version used (Carreira & Zisserman 2017) are no longer available from official site, resulting in some difference of results.
C REDUNDANCY ANALYSIS
To motivate our redundancy reduction approach, we measure and visualize the internal redundancy of well known pretrained networks. We analyze the internal feature maps of existing pre-trained I3D- Inception V2 and R(2+1)D networks on Moments in Time and Kinetics. For each model-dataset pair, extract feature maps for all examples in the validation sets, in both time and channel dimensions, and measure their similarity. In detail, our method consists of the following steps: (1) For a given input, first extract the output feature maps from all convolutional layers in the network at hand. (2) In each layer, we measure the similarity of each feature map to each other with Person's 's correlation coefficient (CC) and root mean squared error (RMSE). We additionally flag feature maps that exhibit high similarity as redundant.
(3) After computing this for the validation sets, we average the values over all examples to obtain mean metrics of redundancy per model and per dataset. We additionally compute the ranges of these values to visualize how much redundancy can vary in model-dataset pair. We present quantitative results Table 12 and show examples of our findings Figure
D VA-RED2 ON LONGER-TRAINIA MODEL
In our experiments, all of our models are trained under a common evaluation protocol for a fair comparison. To balance the training cost and model performance, we use smaller epoch size than the original paper to train our models. For example, authors in (Tran et al. 2018) and (Feichtenhoter 2020), train the R(2+1)D models and X3D models for 188 epochs and 256 epochs respectively to pursue the state-of-the art. However, we only train the models for 120 epochs to largely save the computation resources and training time. However, to rule out the possibility that our base models (i.e., without using Dynamic Convolution) benefit from longer training epochs while our VA-RED2 may not, we conduct an ablation study on the epoch size in Table I3 We can see that our method still shows superiority over the base model in terms of the computational cost and accuracy on the 256-epoch model. Thus we conclude that the effectiveness of our method in achieving higher performance with low computation also holds on the longer-training models.
15

---

Published as a conference paper at ICLR 2021
Frame
Frame
Frame
Video with most redandancy
CC: 0.98 RMST: 0.006
RP: 0.75
Video redundancy
CC: 0.67
RMSE: RP: 0.19
Figure 5: Visualization of the first filters of the first layer of I3D, on examples with most (top) and least (bottom) redundancy in the temporal dimension. We exemplify the results on frames 1, 2 and As can be seen, the video with most redundancy consists of a relatively static video with little movement, and the sets of feature maps from frame to frame harbor heavy similarity. The video with least redundancy consists a gift unwrapping with rapid movement (even in the first few frames) and the corresponding feature maps present visible structural differences from frame to frame. Although in both cases, redundancy is present, it is clear that some examples present much more redundancy than others, thus motivating our input-dependent redundancy reduction approach.
Table 13: Comparison between the performance of VA-RED on 120-epoch X3D model and 256-epoch X3D model. We choose X3D-M as our backbone architecture and set the search space as 2. We train one group of models for 120 epochs and the other for 256 epochs.
120 epochs
256 epochs
Model
Dynamic
GFLOPs
clip-1
vidco-1
vidco-5
GFLOPs
clip-1
vidco-1
video-5
6.42
70.6
90.0
6.42
64.4
72.3
90.8
X3D-M
V
5.38
65.3
72.4
90.7
5.87
66.4
73.6
91.2
E FEATURE MAP VISUALIZATIONS
To further validate initial motivation, we visualize the feature maps which fully computed by the original convlution operation and those which are generated by the cheap operations. We demonstrate those in both temporal dimension (c.f. Figure and channel dimension (c.f. Figure 7 In both cases we can see that the proposed cheap operation generates meaningful feature maps and some of them looks even no difference from the original feature maps.
F POLICY VISUALIZATIONS
To compare with the policy on Mini-Kinetics- (Figure 3 of the main paper), we also visualize the ratio of features which are consumed each layer on Kinetics-400 (c.f. Figure B and Moments- In-Time (c.f. Figure 99. We can see from these two figures that the conclusions we draw from Mini-Kientics-200 still hold. Specifically, In X3D, point-wise convolutions which right after the depth-wise convolutions have more variation among classes and network tends to consume more temporal-wise features at the early stage and compute more channel-wise features at the late stage of the architecture. However, R(2+1)D choose to select fewer features at early stage by both temporal-wise and channel-wise policy. Furthermore, we count the FLOPs of each instance on Kinetics-400, and Moments- In- Time and plot pic charts to visualize the the
Mini- Binii--inetic-222 distribution of this instance-level computational cost. We analyze such distribution with two models: R(2+1 1)D-18 and X3D-M. All of the results demonstrated in Figure 10
16

---

Published as a conference paper at ICLR 2021
G QUALITATIVE RESULTS
We show additional imput examples which consume different levels of computational cost on Kinetics- 400 dataset (c.f. Figure IT and Moments-In- In-is- dataset (c.f. Figure 12). To be consistent, we
Input
Original
feature maps
Our feature maps
Input frames
Original feature maps
Our
feature maps
Figure 6: Visualization of temporal-wise feature maps. We plot the temporal feature maps which are fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps marked with red bounding boxes are cheaply generated. We do this analysis on 8-frame dynamic R(2+1)D-18 pretrained Mini-Kinetics-222 These feature maps are the output of the first spatial convolution combined with ReLU non-linearity inside the ResBle 1 Block_11 We can that most of the cheaply generated feature maps looks difference from the original feature maps, which further support our approach. Best viewed in color.
Input
Original
Our
Input
Original feature maps
Our feature maps
framess
feature maps
feature maps
frames
Input
Original maps
Our maps
Input
Original feature mape
Our
frames
feature maps
Figure Visualization of channel-wise feature maps. We plot the feature maps across the channel dimension. We contrast two kinds of feature maps: fully computed by the original convolution and those mixed with cheaply generated feature maps. The feature maps inside the red bounding boxes cheaply generated. The analysis is performed 8-frame dynamic R(2+1)D-18 18 model which is pretrained on Sini-Kinetics- 200 dataset and we extract these feature maps which are output by the first spatial convolution layer inside the ResBlock
Best viewed in color.
17

---

Published as a conference paper at ICLR 2021
30 Chan wise poliy of XED layors for eoch ostogery
Ei Thrrporal wioh policy of XSD-M layers for cosh conegory
point layers
point layers
R3 Charnel wise pollyy of (D layers each category
k1 Tempontkwio poliyy of RRD+DD MDD+-DD-t byyers ench coteggoy
residuallayers
residualléyers
Figure Ratio computed feature per layer and class on Kinetics-400 dataset. visualize the per-block policy of X3D-M and R(2+1)D-18 on all 400 classes. Lighter color means fewer feature maps are computed while darker color represents more feature maps are computed. While X3D-M tends to consume more temporal- wise features the early stage and compute more channel-wiss features at the late stage, R(2+ I)D choose to select fewer features early stage by both temporal-wise and channel-wise policy. For both architectures, the channel-wise policy has more variation than the temporal-wise policy among different categories.
Channel wise policy X3D for category
Temporal policy of X3D layers each category
point wise layers
point wise layers
(c) Channel-wise policy of R/2+1/D-18 1000-81aaa layers for each category
(d) Temporal wise palicy of R/2+1JD-18 layers each category
residuallayer
layers
Figure 9: Ratio of computed feature per layer and class Moments- In-Time dataset. We visualize the per-block policy of X3D-M and R(2+1)D--1 all 339 classes. Lighter color means fewer feature maps computed while darker color represents more feature maps are computed.
use the 16-frame dynamic R(2+1)D--1 as our pre-trained model. We can see that the examples consuming less computation tend to have less temporal motion, like the second example in Figure or have relatively simple scene configuration, like the first and second examples in Figure 12
18

---

Published as a conference paper at ICLR 2021
J 1LAG
Basioone RE 100-18
Bacibone R2-110-18
Eholaone - 1DD18 Moneria
Datiees's N00a--Grallea--000
Delimeia Kreiics-40
3112320
4ESS
21/
22.4%
Ditioete Mic--orelcca-20
Dalizela Kinalcs--DD 400
Dacasala Moneetia
Figure 10: Computational cost distribution across different models on different datasets. We count the computation of each instance cost different models different datasets. For instance, for the upper-left we use model backbone R(2+1)D- of videos in Sini-Kinetics-ii 200 (Dataset) consuming 38.6 41 GFLOPs by using R(2+1)D-IL (Backbone), 8.8% of videos consuming 35.9 38.6 GFLOPs, and 3.5% of videos consuming 41.4 44.2 GFLOPs.
Mini Mini-Kinetics-
This sub-figure indicates there are 87.7%
More computation: 89.2%
More computation: 78.0%
More computation 70.1%
Lees computation: 57.2%
Less computation: 69 0%
Less computation: 57.2%
washing dishes
tying knot
surfing crowd
Figure 11: Validation video clips from Kinetics-400. For each category, we plot two input video clips which
consume the most and the least computational cost respectively. We infer these video clips with 16-frame dynamic R(2+1)D-18 which pre-trained on Kinetics-400. The percentage the figure indicates the ratio of the actual computational cost 2D convolution to that the original fixed model, Best viewed color.
More computation: 89.2%
More computaton: 89.9%
More computation: 70.1%
Less computation: 6%
Less computation: 61.4%
Typing
punching
skating
Figure 12: Validation video clips from Moments-In= -n-mmmm For each category, plot two input video clips
which consume the most and the least computational cost respectively. We infer these vidco clips with 16-frame dynamic R(2+1)D-BB 18 which is pre-traie trained on Moments In- Time. The percentage in the figure indicates the ratio of the actual computational cost of 2D convolution to that of the original fixed model. Best viewed in color.
19


Reviews:
[{"review_id": "vUV1D1YoC8A", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros:\n1. The motivation of the submission is clear and the writing is easy to follow. The problem being addressed is a practical and important issue in deploying video models to real applications. \n\n2. Experiments are thorough. This submission has done experiments on both action recognition and action localization task. The performance is promising on most video networks. \n\nCons:\n1. This submission proposes to address the problem of slow video inference, but the only metric they report is the Gflops. We know that Gflops is not a good indicator for speed comparison. Sometimes, a small Gflops does not mean fast speed. So I recommend to add a \"speed\" column in every table of this submission, to compare the actual inference speed among the methods. \n\n2. Regarding the searching method, can authors justify what are the differences to X3D? X3D also search the optimal combination of temporal dimension and channel dimension. Are the differences only on adding efficiency loss?\n\n3. The efficiency loss is widely used in searching efficient image models, like in Proxylessnas. What is the contribution and motivation of using it here in searching video models? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "XwDQqDVhDLk", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a novel framework called $\\text{VA-RED}^2$ to reduce spatial and temporal features to be computed for video understanding, which can reduce FLOPs when inferencing the video but remains the performance. \n\nThe authors have done extensive experiments on video action recognition tasks and spatio-temporal action localization task in the area of video understanding. For the video action recognition task, experiments are carried out using Mini-Kinetics-200, Kinetics-400, and Moments-In-time datasets. For the action localization task, J-HMDB-21 dataset is used. Results show that this framework is promising, which reduces the computation but main the performance. \n\n\nQuestion: \n1. X3D-M in the original paper achieved at top-1 74.6 for Kinetics-400 dataset (but in table 4 reports clip-1: 61.8, video-1: 67.9) and FLOPs is 4.73 (6.20 reported in the paper). Can authors explain why there is a difference here? \n\n\nMinor:\n1. Reduce ratio fact for channel-wise dynamic convolution $r=\\frac{1}{2}^{p_c}$ such as in Fig.2 on page 4 and equation 7, equation 8 in supplementary materials on page 13. I think it would make more sense representing it as $(\\frac{1}{2})^{p_c}$. \n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "E_Y-xBEgZjI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\nThe paper presents a framework to reduce internal redundancy in the video recognition model. To do so, given the input frames, the framework predicts two scaling factors to conduct temporal and channel dimension reduction. The remaining part is reconstructed by cheap operations. The authors show that the framework achieves favorable results on several benchmarks.\n\nStrengths\n+ The paper is well written.\n+ Strong quantitative and qualitative results (Consistent improvement over the state-of-the-art baselines).\n+ Solid ablation studies and analysis.\n\nWeakness\n- The core idea of using a light-weight neural module to maximize the video model efficiency is not novel.\nFor example, AR-Net already has shown that a policy network can decide the video input resolution, i.e., spatial dimension, adaptively, leading to improve both efficiency and accuracy. In this work, the key concept is basically the same and only the primitive operation unit, i.e., temporal and channel dimensions, is different.\n\n- Please provide experiment-level comparisons with AR-Net\n\n- The following SOTA video models are missing in the reference.\nDynamoNet: Dynamic action and motion network, ICCV 2019\nVideo Modeling with Correlation Networks, CVPR 2020\nRubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition, ECCV2020\nDirectional Temporal Modeling for Action Recognition, ECCV2020\n\nQuestion\n\n- Can the idea be applied to dense video understanding tasks, such as video semantic segmentation?\n\nRating\n\n- Aiming at acquiring an efficient model in a data-driven manner is indeed important for video models.\nWhile the key idea is not novel, the paper has obvious empirical contributions that may help communities to invigorate future researches in this direction. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on efficient video analysis.  The reviewers appreciated the clear formulation and effective methodology.  Concerns were raised over empirical validation.  The authors' responses added additional material that assisted in clarifying these points.  After the discussion the reviewers converged on a unanimous accept rating.  The paper contains solid advances in efficient inference for video analysis and is ready for publication.
Decision: Accept (Poster)