Look More but Care Less in Video Recognition
Authors: Yitian Zhang
Yue Bai
Huan Wang
Yi Xu
Yun Fu
Venue: NeurIPS 2022 Accept (2022)
Keywords: Action Recognition
Dynamic Networks
Efficiency

Abstract: Existing action recognition methods typically sample a few frames to represent each video to avoid the enormous computation, which often limits the recognition performance. To tackle this problem, we propose Ample and Focal Network (AFNet), which is composed of two branches to utilize more frames but with less computation. Specifically, the Ample Branch takes all input frames to obtain abundant information with condensed computation and provides the guidance for Focal Branch by the proposed Navigation Module; the Focal Branch squeezes the temporal size to only focus on the salient frames at each convolution block; in the end, the results of two branches are adaptively fused to prevent the loss of information. With this design, we can introduce more frames to the network but cost less computation. Besides, we demonstrate AFNet can utilize less frames while achieving higher accuracy as the dynamic selection in intermediate features enforces implicit temporal modeling. Further, we show that our method can be extended to reduce spatial redundancy with even less cost. Extensive experiments on five datasets demonstrate the effectiveness and efficiency of our method.

Look More but Care Less in Video Recognition
Yitian Zhang Yue Bai' Huan Wang' Xu Yun Fu Fu
Department of Electrical and Computer Engineering, Northeastem University "Khoury College of Computer Science, Northeastern University
Abstract
Existing action recognition methods typically sample few frames represent each video to avoid the enormous computation, which often limits the recognition performance. To tackle this problem, propose Ample and Focal Network (AFNet), which is composed of two branches to utilize more frames but with less compulation. Specifically, the Ample Branch takes all input frames to obtain abundant information with condensed computation and provides the guidance for Focal Branch by the proposed Navigation Module; the Focal Branch squeezes the temporal size to only focus on the salient frames at each convolution block; in the end, the results of branches are adaptively fused to prevent the loss of information. With this design, we can introduce more frames to the network but cost less computation. Besides, we demonstrate AFNet can utilize fewer frames while achieving higher accuracy as the dynamic selection in intermediate features enforces implicit temporal modeling. Further, we show that our method can be extended to reduce spatial redundancy with even less cost. Extensive experiments on five datasets demonstrate the effectiveness and efficiency of our method. Our code is available at
Introduction
Online videos have grown wildly in recent years and video analysis is necessary for many applications
such as recommendation [6]. surveillance [4, 5] and autonomous driving [31, 17]. These applications require not only accurate but also efficient video understanding algorithms. With the introduction of
deep learning networks [3] in video recognition, there has been rapid advancement in the performance
of the methods in this area. Though successful, these deep learning methods often cost huge computation, making them hard to be deployed in the real world.
In video recognition, we need to sample multiple frames represent each video which makes the computational cost scale proportionally to the number of sampled frames. most cases, a small proportion of all the frames is sampled for each input, which only contains limited information of the original video. A straightforward solution is to sample more frames to the network but the computation expands proportionally to the number of sampled frames.
There are some works proposed recently to dynamically sample salient frames [29, 16] for higher efficiency. The selection step of these methods is made before the frames are sent to the classification network, which means the information of those unimportant frames is totally lost and it consumes a considerable time for the selection procedure. Some other methods proposed to address the spatial redundancy in action recognition by adaptively resizing the resolution based on the importance of each frame [23], or cropping the most salient patch for every frame [28]. However, these methods still completely abandon the information that the network recognizes as unimportant and introduce a policy network make decisions for each sample which leads to extra computation and complicates the training strategics.
Corresponding Author: markcheung9248 @ @gmail.mmm....m.......em
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Zoupanal Setectio
Res Block
Ample Branch
Block
Cone
Backbone
Existing paraddigms to sulve tempural redundancy
Aospanal
Cone
Conv
AF Modull
Module
Module
Block
Ris
Nefeclion
Focal Beanch
Backbone
AFNet
AF Medule
Figure 1: Comparisons between existing methods and proposed Ample and Focal Network (AFNet). Most existing works reduce the redundancy in data at the beginning of the deep networks which leads to the loss of information. We propose two-branch design which processes frames with different computational resources within the network and preserves all input information as well.
In our work, we from another perspective compared with previous works. We propose a method which makes frame selection within the classification network. Shown in Figure 1, we design an architecture called Ample and Focal Network (AFNet) which is composed of two branches: the ample branch takes glimpse of all the input features with lightweight computation as we downsample the features for smaller resolution and further reduce the channel size; the focal branch receives the guidance from the proposed navigation module to squeeze the temporal size by only computing on the selected frames to save cost; in the end, we adaptively fuse the features of these two branches to prevent the information loss of the unselected frames.
In this manner, the two branches are both very lightweight and we enable AFNet to look broadly by sampling more frames and stay focused on the important information for less computation. Considering these two branches in a uniform manner, on the one hand, we can avoid the loss of information compared other dynamic methods as the ample branch preserves the information of all the input: on the other hand, we can restrain the noise from the unimportant frames by deactivating them in each convolutional block. Further, we have demonstrated that the dynamic selection strategy at intermediate features is beneficial for temporal modeling it implicitly implements frame-wise attention which can enable our network to utilize fewer frames while obtaining higher accuracy. In addition, instead of introducing policy network to select frames, we design lightweight navigation module which can be plugged into the network so that our method can easily be trained in an end-to- end fashion. Furthermore, AFNet is compatible with spatial adaptive works which can help to further reduce the computations of our method.
We summarize the main contributions as follows:
We propose an adaptive two-branch framework which enables 2D-CNNs to process more frames with less computational cost. With this design, we not only prevent the loss of information but strengthen the representation of essential frames.
We propose a lightweight navigation module to dynamically select salient frames each convolution block which can easily be trained in an end-to-end fashion.
The selection strategy at intermediate features not only empowers the model with strong flexibility as different frames will be selected at different layers, but also enforces implicit temporal modeling which enables AFNet to obtain higher accuracy with fewer frames.
We have conducted comprehensive experiments on five video recognition datasets. The results show the superiority of AFNet compared to other competitive methods.
2 Related Work
2.1 Video Recognition
The development of deep learning in recent years serves as a huge boost to the research of video recognition. A straightforward method for this task is using 2D-CNNs to extract the features of sampled frames and use specific aggregation methods to model the temporal relationships across frames. For instance, TSN [27] proposes to average the temporal information between frames. While TSM [20] shifts channels with adjacent frames to allow information exchange at temporal dimension. Another approach is to build 3D-CNNs to for spatiotemporal learning, such C3D [26], I3D [3] and SlowFast [8]. Though being shown effective, methods based on 3D-CNNs are computationally expensive, which brings great diffieulty in real-world deployment.
While the two-branch design has been explored by SlowFast, our motivation and detailed structure different from it in the following ways: 1) network category: SlowFast is static 3D model, but

---

AFNet is dynamic 2D network; 2) motivation: SlowFast aims to collect semantic information and changing motion with branches at different temporal speeds for better performance, while AFNet is aimed to dynamically skip frames to save computation and the design of two-branch structure is to prevent the information loss; 3) specific design: AFNet is designed to downsample features for efficiency at ample branch while SlowFast processes features in the original resolution; 4) temporal modeling: SlowFast applies 3D convolutions for temporal modeling, AFNet is a 2D model which is enforced with implicit temporal modeling by the designed navigation module.
2.2 Redundancy in Data
The efficiency of 2D-CNNs has been broadly studied in recent years. While some of the works aim at designing efficient network structure [13], there is another line of research focusing on reducing the intrinsic redundancy in image-based data [32, 11]. In video recognition, people usually sample limited number of frames to represent each video to prevent numerous computational costs. Even though, the computation for video recognition is still a heavy burden for researchers and a common strategy to address this problem is reducing the temporal redundancy in videos as not all frames are essential to the final prediction. [33] proposes to use reinforcement learning to skip frames for action detection. There are other works [29, 16] dynamically sampling salient frames to save computational cost. As spatial redundancy widely exists in image-basee data, [23] adaptively processes frames with different resolutions. [28] provides the solution cropping the most salient patch for each frame. However, the unselected regions or frames of these works are completely abandoned. Hence, there will be some information lost in their designed procedures. Moreover, most of these works adopt policy network to make dynamic decisions, which introduces additional computation somehow and splits the training into several stages. In contrast, our method adopts a two-branch design, allocating different computational resources based on the importance of each frame and preventing the loss of information. Besides, design a lightweight navigation module to guide the network where look, which can be incorporated into the backbone network and trained in an end-to-end way. Moreover, we validate that the dynamic frame selection intermediate features will not only empower the model with strong flexibility as different frames will be selected at different layers, but result in learned frame-wise weights which enforce implicit temporal modeling.
3 Methodology
Intuitively, considering more frames enhances the temporal modeling but results in higher computa- tional cost. To efficiently achieve the competitive performance, we propose AFNet to involve more frames but wisely extract information from them to keep the low computational cost. Specifically, we design a two-branch structure to treat frames differently based on their importance and process the features in adaptive manner which can provide our method with strong flexibility. Besides, we demonstrate that the dynamic selection of frames in the intermediate features results in learned frame-wist weights which can be regarded as implicit temporal modeling.
3.1 Architecture Design
As is shown in Figure 2, we design our Ample and Focal (AF) module as a two-branch structure: the ample branch (top) processes abundant features of all the frames in a lower resolution and a squeezed channel size; the focal branch (bottom) receives the guidance from ample branch generated by the navigation module and makes computation only on the selected frames. Such design can be conveniently applied to existing CNN structures to build AF module.
Ample Branch. The ample branch is designed to involve all frames with cheap computation, which serves as 1) guidance to select salient frames to help focal branch concentrate on important information; 2) complementary stream with focal branch to prevent the information loss via carefully designed fusion strategy.
Formally, we denote video sample i as vf, containing T frames as of = fi..,,.. For convenience, we omit the superscript in the following sections if no confusion arises. We denote the input of ample branch as us E R XX''XCXXXXX xW where C' represents the channel size and H x W is the spatial size. The features generated by the ample branch can be written as:
Vyn F (Va).
(I)
3

---

Ample Branch
Iwi
Madule
Medule
I Tanguril Mack
Temperal Misk
Dynamic Fasion
Focal Branch
Additoon
Muttipiiation
Dosmnsmpll
f Upsample
Vanilla Convolution
Sparsc Convolation
Connectio
Figure 2: Architecture of AF module. The module is composed of two branches, the ample branch would process all the input features in lower resolution and reduced channel size; while the focal branch would only compute the features of salient frames (colored features) guided by our proposed navigation module. The results of two branches adaptively fused at the end of AF module that we can prevent the loss of information.
where Bye E TPXXCC./)XX(B.//)XX.... represents the output of ample branch and F" stands for series of convolution blocks. While the channel, height, width at focal branch are denoted as Co, H,, Wo correspondingly. We set the stride of the first convolution block to 2 to downsample the resolution of this branch and we upsample the feature the end of this branch by nearest interpolation.
Navigation Module. The proposed navigation module is designed to guide the focal branch where to look by adaptively selecting the most salient frames for video v².
Specifically, the navigation module generates binary temporal mask Ln using the output from the n-th convolution block in ample branch Bye. At first, average pooling is applied to Bye to resize the spatial dimension to x 1, then we perform convolution to transform the channel size to
Byg ReLU (BN (Wi Pool (U)))))
where stands for convolution and W, denotes the weights of the 1 convolution. After that, we reshape the dimension of feature uyg from T x 1 x 1 to 1 x (2 7) x 1 x 1 so that we can model the temporal relations for each video from channel dimension by:
p/
W2 yp
where W2 represents the weights of the second 1 convolution and it will generate a binary logit P/ E IR2 for each frame which denotes whether to select it.
However, directly sampling from such discrete distribution is non-differentiable. In this work, we apply Gumbel-Softmax [14] to resolve this non-differentiability. Specifically, we generate a normalized categorical distribution by using Softmax:
exp pil
1/ 1144 exp (pl)) + cxp (p²tt S
(4)
and we draw discrete samples from the distribution π as:
L arg max (log Iy + G,),
(5)
where G, logllllloo log U,) is sampled from Gumbel distribution and U, is sampled from Unif(0,1) which is a uniform distribution. As arg max cannot be differentiated, we relax the discrete sample L in backpropagation via Softmax:
exp ((ogg, + GG)
i,
(6)
Σ exp (logg4 + G2)
the distribution i will become one-hot vector when the temperature factor 7 + 0 and we let decrease from to 0.01 during training.
Focal Branch. The focal branch is guided by the navigation module to only compute the selected frames, which diminishes the computational cost and potential noise from redundant frames.

---

The features at the n-th convolution block in this branch can be denoted as oo BBT Based on the temporal mask I. generated from the navigation module, we select frames which have corresponding non-zero values the binary mask for each video and apply convolutional operations only on these extracted frames v E [g [IlixxoXH.XW..
' = Fl (1'
(7)
where F.f is the n-th convolution blocks at this branch and we set the group number of convolutions to 2 in order further reduce the computations. After the convolution operation n-th block, generate a zero-tensor which shares the same shape with y and fill the value by adding and " " with the residual design following [12].
y/
At the end of these two branches, inspired by [1, 11], we generate a weighting factor by pooling and linear layers to fuse the features from two branches:
U/ о Ey- + (1 0) O 0gg.
where C denotes the channel-wise multiplication.
3.2 Implicit Temporal Modeling
While our work is mainly designed to reduce the computation in video recognition like [28, 24], we demonstrate that AFNet enforces implicit temporal modeling by the dynamic selection of frames in the intermediate features. Considering TSN[27] network which adapts vanilla ResNet[ [2] structure, the feature at the n-th convolutional block each stage can be written as Un IR TxCCHX Thus, the feature at n + 1-th block can be represented as:
Un+1 + Fu+1 (Un)
= (1 + Avn+ 1) Uas
Fxt1 (vn)
Ann+1
(10)
Un
where Fo+1 is the n + 1-th convolutional block and we define Ae as the coefficient learned from this block. By that we can write the output this stage UN as:
NV
UN
II (1+
(v))
(11)
n
Similarly, we define the features in ample and focal branch as:
NN
II (1 + Auya) * Uyr
(12)
N
(1 Ten * + se0z) Pgg
II n=
(13)
U
where Ln is the binary temporal mask generated by Equation and By denotes the input this stage. Based on Equation 8, we can get the output of this stage as:
Byn = + Uy + (1 0) о v
(14)
II (1+ (1+Aag)) Auy;) L
+ 0)
mnd I1 Ln=2
* Un
As Ln a temporal-wis. binary mask, will decide whether the coefficient Au, will be calculated in each frame at every convolutional block. Considering the whole stage is made up of multiple convolutional blocks, the series multiplication of focal branch's output with the binary mask L,, will approximate soft weights. This results in learned frame-win wise weights each video which we regard as implicit temporal modeling. Although we do not explicitly build any temporal modeling module, the generation of T., in Equation has already taken the temporal information into account so that the learned temporal weights equal performing implicit temporal modeling at cach stage.
5

---

Selection
Figure Illustaaioon of extending AFNet reduce spatial redundancy further improve the efficiency. Only the colored area will be calculated at the inference stage.
3.3 Spatial Redundancy Reduction
In this part, show that our approach is compatible with methods that aim to solve the problem of spatial redundancy. We extend the navigation module by applying similar procedures with the
temporal mask generation and the work [11] to generate spatial logit for the n-th convolution block which is shown in Figure 3:
d = W4 * (Pool (ReL.U (BN (W3 lyy:))))
(15)
where W3 denotes the weights of the x 3 convolution and W4 stands for the weights convolution with kernel size x 1. After that, still use Gumbel-Softmat to sample from discrete distribution to generate spatial mask M,, and navigate the focal branch to merely focus on the salient regions of the selected frames further reduce the cost.
3.4 Loss functions
Inspired by [27], we take the average of each frame's prediction represent the final output of the corresponding video and our optimization objective is minimizing:
- y log (P (v) + Σ (r RT)²
(16)
(v.y)
The first term the cross-entropy between predictions P (v) for input video u the corresponding one-hot label We denote in the second term as the ratio of selected frames in every mini-batch and RT as the target ratio which is set before the training (RS is the target ratio when extending navigation module to reduce spatial redundancy). We let approximate RT by adding the second loss term and manage the trade-off between efficiency and accuracy by introducing a factor À which balances these two terms.
4 Empirical Validation
In this section, we conduct comprehensive experiments to validate the proposed method. We first compare method with plain 2D CNNs to demonstrate that our AF module implicitly implements temporal-wise attention which is beneficial for temporal modeling. Then, we validate AFNet's efficiency by introducing more frames but costing less computation compared with other methods. Further, we show AFNet's strong performance compared with other efficient action recognition frameworks. Finally, provide qualitative analysis and extensive ablation results demonstrate the effectiveness of the proposed navigation module and two-branch design.
Datasets. Our method is evaluated on five video recognition datasets: (1) Mini-Kinetich [23, 24] is subset Kinetics [15] which selects 200 classes from Kineties, containing 121k training videos and 10k validation videos; (2) ActivityNet-vl. 1.3 [2] is an untrimmed dataset with 200 action categories and average duration of 117 seconds. It contains 10,024 video samples for training and 4,926 for validation; (3) Jester is hand gesture recognition dataset introduced by [22]. The dataset consists 27 classes, with I 19k training videos and 15k validation videos; (4) Something-SSmettingg V1&V2 [10] are two human action datasets with strong temporal information, including 98k and 194k videos for training and validation respectively.
Data pre-processing. We sample 8 frames uniformly to represent every video on Jester, Mini- Kinetics, and 12 frames on ActivityNet and -eeethingSSmmthingg to compare with existing works unless specified. During training, the training data is randomly cropped to 224 x 224 following [35], and we perform random flipping except for Something-Somen Something. At inference stage, all frames are center-cropped to 224 x 224 and one-crop one-clip per video for efficiency.

---

Implementation details. Our method is bulit on ResNet50 [12] in default and we replace the first three stages of the network with our proposed AF module. We first train our two-branch network from scratch on ImageNet for fair comparisons with other methods. Then we add the proposed navigation module and train along with the backbone network on video recognition datasets. In our implementations, RT denotes the ratio of selected frames while RS represents the ratio of select regions which will decrease from I to the number we set before training by steps. We let the temperature in navigation module decay from to 0.01 exponentially during training. Due to limited space, we include more details of implementation supplementary material.
4.1 Comparisons with Existing Methods
Less is more. At first, we implement AFNet on Nomething-Something
Table 1: Comparisons with baseline method on Something-Something V1 and Jester datasets.
VI and Jester datasets with 8 sampled frames. We com- pare with the baseline method TSN as both methods do not explicitly build temporal modeling module and are built ResNet50. In Table I, our method AFNe:(RT=1100) shows similar performance with TSN when selecting all the frames. Nonetheless, when we select fewer frames AFNet, exhibits much higher accuracy compared to TSN and AFNet(RT=1.00 .00 which achieves Less More by utilizing less frames but obtaining higher accuracy. The results may seem counterintuitive as seeing more frames is usually beneficial for video recognition. The explanation is that the two-branch design of AFNet can preserve the information of all input frames and the selection of salient frames intermediate features implements implicit temporal modeling as we have analyzed Section 3.2. As the binary mask learned by the navigation module will decide whether the coefficient will be cal- culated for each frame at every convolutional block, it will result learned temporal weights each video. To better illustrate this point, conduct the experiment by removing Cumbel-Softmax [14] in our navigation module and modifying it to learn soft temporal weights for the features at focal branch. We can observe that (oftessoft-wigght has similar performance with AFNet(RT-0.25) AFNeI(RT=0.50 and outperforms AFNet(RT=1100) significantly which indicates that learning soft frame-wise weights causes the similar effect.
Sth-Sth V1
Jester
Method
Frame
Top-I Acc.
Top-I Acc.
TSN [our imp)
8
18.6%
83.5%
AFNet (KT=1 1.00).
8
19.2%
83.6%
AFNet (kT=).50)
8
26.8%
89.2%
AFNet (KT=0.25)
8
27.7%
89.2%
AFNet (soft-weights
8
27.0%
89.9%
Table Performance comparison on Samething-Somethiee (Sth-Sth) datasets. GFLOPs represents the average computation to process one video.
Sth-Sth
Sth-Sth
Method
Dynamic
Backbone
Frames
Top-1 Acc.
GFLOPs
Top-1 Acc.
GFLOPs
TRN RRN/Ce/Fi [34]
X
BN-Inception
8/8
42.0%
32.0
55.5%
32,0
ECO [35]
X
BN-IncepiioaRR<Nee1168
8
39.6%
32.0
TSM
X
ReaNet50
8
45.6%
32.7
59.1%
32.7
bLVNet-TAM
X
bLRessNe500
16
47.8%
60.2%
TANet [2]]
X
ResNet50
8
47.3%
33.0
60.5%
33.0
SmallBig [18]
X
ResNer50
8
47.0%
52.0
59.7%
$2.0
TEA [19]
X
ResNet50
8
48.9%
35.0
60.9%
35.0
SlowFast [8]
ResNec50-ResNe50
8x8
61.7%
66.6x3
AdaFuse-TSM [24]
ResNet50
8
46.8%
31.5
59,8%
31.3
AdaFocus-TSM [28]
MobileNeaV2-ResNee50
8+12
48.1%
33.7
60.7%
33.7
AFNet-TSM (KT=01.4
AF-ResNet50
12
49.0%
27.9
61.3%
27.8
AFNes-TSM (RT=08)
Al-ResNet50
12
49.9%
62.5%
31.7
AFNet-TSM (RT=0.4)
AF-MobileNerV3
12
45.3%
2.2
58.4%
2.2
AFNet-TSM (RT=08))
AF-MobileNe1V3
12
45.9%
2.3
58.6%
2.3
AFNet-TSM (RT=0.4)
Al-ResNeri01
12
49.8%
42.1
62.5%
41.9
AFNet-TSM (RT+
AF-ResNetl0l
12
50.1%
63.2%
48.5
More is less. We incorporate our method with temporal shift module (TSM [20]) to validate that A can further reduce the redundancy of such competing methods and achieve More is Less by seeing more frames with less computation. We implement our method on Something-Somethiee VI&V2 datasets which contain strong temporal information and relevant results are shown in Table 2.
7

---

Table 3: Comparisons with competitive efficient video recognition methods Mini-Kinetics. AFNet achieves the best trade-off compared to existing works. GFLOPs represents the average computation to process one video.
Table 4: Comparisons with competitive efficient video recognition methods on ActivityNet. AFNet achieves the best trade-off compared to existing works. GFLOPs represents the average computation to process one video.
Mini-Kinetics
ActivityNet
Method
Method
Top-1 1 Acc.
GFLOPs
mAP
GFLOPs
LiteEval [30]
61.0%
99.0
AdaFrame [29]
71.5%
79.0
SCSampler [16]
70.8%
42.0
LiteEval [30]
72.7%
95.1
AR-Net [23]
71.7%
32.0
ListenToLook [9]
72.3%
81.4
AdaFuse [24]
72.3%
23.0
SCSampler [16]
72.9%
42.0
AdaFocus [28]
72.2%
26.6
AR-Net [23]
73.8%
33.5
VideolQ [25]
72.3%
20.4
VideolQ [25]
74.8%
28.1
AdaFocus [28]
75.0%
26.6
AFNet (RT=0,4)
72.8%
19.4
AFNet (RT-0.8)
73.5%
22.0
AFNet (RS=0.4.RT-0.8)
75.6%
24.6
0.9
76
1.44x
Ratio 0.80
0.8
Ratio
Ratio 0.25
74
1.884
0.7
MAP (1%)
72
0.6
0.5
70
Adafocus
ARNet
0.4
68
SCSampler
LisennbLook
0.3
66
0.2
15
20
25
30
35
40
45
50
3
4
7 8
10
GFLOPs
Bottleneck Block Index
Figure 4: Computational cost vs mean Average Precision on ActivityNet. Our method is imple- mented with different ratio of selected frames RT E (0.6, 0.8, 1.0) and fixed ratio of selected regions RS E {0.4).
Figure 5: Distribution of RT E {0.25, 0.8) for each convolution block AF-ResNet50 on Something-Somettiee
V1. Dash lines denote us- ing 3rd-order polynomials estimate the trend of distribution.
Compared to TSM which samples 8 frames, our method shows significant advantages in performance as we introduce more frames and the two-branch structure can preserve the information of all frames. Yet, our computational cost is much smaller than TSM as we allocate frames with different computation resources by this two-branch design and adaptively skip the unimportant frames with the proposed navigation module. Moreover, AFNet outperforms many static methods, which carefully design their structures for better temporal modeling. both in accuracy and efficiency. This can be explained by that the navigation module restrains the noise of unimportant frames and enforces frame-wii wise attention which is beneficial for temporal modeling. As for other competitive dynamic methods like AdaFuse and AdaFocus, our method shows obviously better performance both in accuracy and computations. When costing similar computation, AFNet outperforms AdaFuse and AdaFocus by 3.1% and 1.8% respectively our method on other backbones for even higher accuracy and efficiency. When we build AFNet on efficient structure MobileNetV3, computation of 2.3 GFLOPs. Besides, PPNNe-TSM(RT-D))
Sammething-Sometthnee V1. Furthermore, we implement
obtain similar performance with TSM but only with the
with the backbone ResNet101 can achieve the accuracy of 50.1% % and 63.2% on Something- Smmething-SSmethingg VI and V2, respectively, which further validate the effectiveness and generalization ability of our framework.
Comparisons with competitive dynamic methods. Then, we implement our method on Mini- Kinetics and ActivityNet, and compare AFNet with other efficient video recognition approaches. At first, we validate our method on Mini-Kinetics and AFNet shows the best performance both in accuracy and computations compared with other efficient approaches in Table To demonstrate that AFNet can further reduce spatial redundancy, extend the navigation module to select salient

---

(a) Tearing something just little bit
(b) Dropping something front of something
(c) Pulling something from right left
Figure 6: Visualization results on crosses denote the ones that AFNet does select.
V1 dataset. Frames annotated with red
Table 5:
Ablation study on navigation module and two-branch architecture. Different sampling policies are compared with our dynamic temporal selection module in various selection ratios.
Ablation
mAP
Structure
Temporal Sampling Policy
Spatial Sampling Policy
0.25
0.50
0.75
Single Branch
Navigation Module
46.8%
60.4%
71.3%
Random Sampling
62.8%
71.3%
74.2%
Two Branch
Uniform Sampling
59.0%
71.7%
74.7%
Normal Sampling
61.5%
71.4%
74.6%
Two Branch
Navigation Module
72.2%
74.3%
75.8%
Random Sampling 69.2%
71.4%
74.1%
Two Branch
Center Cropping 68.9%
72.0%
73.9%
Two Branch
Navigation Module
72.0%
74.3%
75.9%
regions of important frame ActivityNet. We move the temporal navigation module to the first layer of the network to avoid huge variance features when incorporating spatial navigation module and note that only apply this procedure in this part. We obtains the best performance while costing the least computation compared to other works. Moreover, change the ratio of selected frames and plot the mean Average Precision and computational cost of various methods in Figure 4. We can conclude that AFNet exhibits a better trade-off between accuracy and efficiency compared to other works.
from Table 4 that method
4.2 Visualizations
We show the distribution of RT among different convolution blocks under different selection ratios in Figure 5 and utilize 3rd-order polynomials to display the trend of distribution (shown in dash lines). One can see a decreased trend RT for all the curves with the increased index in convolution blocks and this can be explained that earlier layers mostly capture low-level information which has relatively large divergence among different frames. While high-level semantics between different frames are more similar, therefore AFNet tends to skip more later convolution blocks. In Figure we visualize the selected frames the 3rd-block of our AFNet with RT=0.5 on the validation set of Something-SSe Sombtiiing V1 where uniformly sample frames. Our navigation module effectively guides the focal branch to concentrate on frames which are more task-relevant and deactivate the frames that contain similar information.
4.3 Ablation Study
In this part, we implement our method on ActivityNet with 12 sampled frames to conduct compre- hensive ablation study to verify the effectiveness of our design.
Effect of two branch design. We first incorporate our navigation module into ResNet50 and compare it with AFNet to prove the strength of our designed two-branch architecture, From Table 5, AFNet shows substantial advantages accuracy under different ratios of select frames. Aside from it, models which adopted our structure but with a fixed sampling policy also show significantly better performance compared with the network based on single branch which can further demonstrate the effectiveness of our two-branch structure and the necessity to preserve the information of all frames.

---

Effect of navigation module. In this part, we further compare our proposed navigation module with three alternative sampling strategies in different selection ratios: (1) random sampling; (2) uniform sampling: sample frames in equal step; (3) normal sampling: sample frames from standard gaussian distribution. Shown Table our proposed strategy continuously outperforms other fixed sampling policies under different selection ratios which validates the effectiveness of the navigation module. Moreover, the advantage of our method is more obvious when the ratio of selected frames is small which demonstrates that our selected frames are more task-relevant and contain essential information for the recognition. Further, we evaluate the extension of navigation module which can reduce spatial redundancy, and compare it with: (I) random sampling; (2) center cropping. Our method shows better performance compared with fixed sampling strategies under various selection ratios which verifies the effectiveness of this design.
Conclusion
This paper proposes adaptive Ample and Focal Network (AFNet) to reduce temporal redundancy in videos with the consideration of architecture design and the intrinsic redundancy in data. Our method enables 2D-CNNs have access more frames to look broadly but with less computation by staying focused on the salient information. AFNet exhibits promising performance as our two- branch design preserves the information of all the input frames instead of discarding part of the knowledge at the beginning of the network. Moreover, the dynamic temporal selection within the network not only restrains the noise of unimportant frames but enforces implicit temporal modeling as well. This enables AFNet to obtain even higher accuracy when using fewer frames compared with static method without temporal modeling module. We further show that our method can be extended to reduce spatial redundancy by only computing important regions of the selected frames. Comprehensive experiments have shown that our method outperforms competing efficient approaches both in accuracy and computational efficiency.
Acknowledgments and Disclosure of Funding
Research sponsored by the DEVCOM Analysis Center and was accomplished under Cooperative Agreement Number W91 INF-22-2-0001. The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed or implied, of the Army Research Office or the U.S. Government. The U.S. Government is authorized reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein.
References
[1] Y. Bai, L. Wang, Z. Tao, S. Li, and Y. Fu. Correlative channel-aware fusion for multi-view time series classification. In AAAI, 2021.
[2] Caba Heilbron, V. Escorcia, Ghanem, and J. Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
[3] Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
[4] J. Chen, K. Li, Q. Deng, K. Li, and S. Y. Philip. Distributed deep learning model for intelligent video surveillance systems with edge computing. IEEE Transactions Industrial Informatics, 2019.
[5] R. T. Collins, A. Lipton, and T. Kanade. Introduction to the special section on video surveillance. IEEE Transactions on pattern analysis and machine intelligence, 22(8):745-746, 2000.
[6] J. Davidson, B. Liebald, J. Liu, P. Nandy, T. Van Vleet, U. Gargi, S. Gupta, Y. He, M. Lambert, B. Livingston, et al. The youtube video recommendation system. In ACM conference on Recommender systems, 2010.
10

---

[7] Q. Fan, C.-F. Chen, H. Kuehne, M. Pistoia, and D. Cox. More is less: Learning efficient video representations by big-little network and depthwise temporal aggregation. arXiv preprint arXiv: 2019.
[8] C. Feichtenhofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. In ICCV, 2019.
[9] R. Gao, T.-H. Oh, K. Grauman, and L. Torresani. Listen to look: Action recognition by previewing audio. In CVPR, 2020.
[10] R. Goyal, S. Ebrahimi Kahou, V. Michalski, J. Materzynska, S. Westphal, H. Kim, V. Haenel,
I. Fruend, P. Yianilos, M. Mueller-Freitage et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
[11] Y. Han, G. Huang, S. Song, L. Yang. Y. Zhang, and H. Jiang. Spatially adaptive feature refinement for efficient inference. IEFE Transactions on Image Processing, 30:9345-9358, 2021.
[12] K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In CVPR, 2016.
[13] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv; 770.00461, 2017.
[14] E. Jang, S. Gu, and B. Poole. Categorical reparametcrization with gumbel-softmax. arXiv preprint arXiv: 161 144, 2016.
[15] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 1705. 2017.
[16] B. Korbar, D. Tran, and L. Torresani. Scsampler: Sampling salient clips from video for efficient action recognition. In ICCV, 2019.
[17] P. Li and J. Jin. Time3d: End-to-end joint monocular 3d object detection and tracking for autonomous driving. CVPR, 2022.
[18] X. Li, Y. Wang, Z. Zhou, and Y. Qiao. Smallbignet: Integrating core and contextual views for video classification. In CVPR, 2020.
[19] Y. Li. B. Ji, X. Shi, J. Zhang. B. Kang, and L. Wang. Tea: Temporal excitation and aggregation for action recognition. In CVPR, 2020.
[20] J. Lin, C. Gan, and S. Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019.
[21] Z. Liu, L. Wang. W. Wu, C. Qian, and T. Lu. Tam: Temporal adaptive module for video recognition. In CVPR, 2021.
[22] J. Materzynska, G. Berger, I. Bax, and R. Memisevic. The jester dataset: A large-scale video dataset of human gestures. In ICCVW, 2019.
[23] Y. Meng, C.-C. Lin, Panda, P. Sattigeri, L. Karlinsky, A. Oliva, K. Saenko, and R. Feris. Ar-net: Adaptive frame resolution for efficient action recognition. In ECCV, 2020.
[24] Y. Meng, R. Panda, C.-C. Lin, P. Sattigeri, L. Karlinsky, K. Saenko, A. Oliva, and R. Feris. Adafuse: Adaptive temporal fusion network for efficient action recognition. arXiv preprint arXiv:2102.05775, 2021.
[25] X. Sun, R. Panda, C.-F. R. Chen, A. Oliva, R. Feris, and K. Saenko. Dynamic network quantization for efficient video inference. In ICCV, 2021.
[26] D. Tran, I.. Bourdev, Fergus, L. Torresani, and Paluri. Learning spatiotemporal features with 3d convolutional networks. In ICCV, 2015.

---

[27] L. Wang, Y. Xiong, Z. Wang, Y. Qiao, D. Lin, X. Tang, and L. Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
[28] Y. Wang, Z. Chen, H. Jiang, S. Song, Y. Han, and G. Huang. Adaptive focus efficient video recognition. arXiv preprini arXiv:2105.0322 2021.
[29] Z. Wu, H. Li, C. Xiong, Y.G. Jiang, and L. S. Davis. dynamic frame selection framework for fast video recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
[30] Z. Wu, C. Xiong, Y.G. Jiang, and L. S. Davis. Liteeval: A coarse-to-fine framework for resource efficient video recognition. arXiv preprint arXiv:1912.01601, 2019.
[31] Y. Xu, L. Wang, Y. Wang, and Y. Fu. Adaptive trajectory prediction via transferable gnn. In CVPR, 2022.
[32] L. Yang, Y. Han, X. Chen, S. Song, J. Dai, and G. Huang. Resolution adaptive networks for efficient inference. In CVPR, 2020.
[33] S. Yeung, O. Russakovsky, G. Mori, and L. Fei-Fei. End-to-end learning of action detection from frame glimpses in videos. In CVPR, 2016.
[34] B. Zhou, A. Andonian, A. Oliva, and A. Torralba. Temporal relational reasoning in videos. In ECCV, 2018.
[35] M. Zolfaghari, K. Singh, and T. Brox. Eco: Efficient convolutional network for online video understanding. In ECCV, 2018.
12

---

Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], or [N/A] - You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes] See Section ??.
Did you include the license the code and datasets? [No] The code and the data are proprietary.
Did you include the license to the code and datasets? [N/A]
Please do modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below,
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes] See Section 4.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] See Section 4.
(b) Did you mention the license of the assets? [Yes] See Section 4.
(c) Did you include any assets either in the supplemental material
URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 'rere using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
13


Reviews:
[{"review_id": "yiTSSBEOtH-", "reviewer": "Reviewer_bMpQ", "summary": "The paper proposed to reduce the cost and boost the accuracy of CNN models applied to video action recognition via a two-stream approach. The first, \"ample\" stream processes all of the frames, but cheaply, by using low spatial resolution and number of channels. The second \"focal\" stream processes frames at high-resolution, but only processes a few of the input frames. A navigation model uses the input from the ample stream to select which frames the focal stream should process, using Gumbel softmax.\n\nThe paper presents results on the ActivityNet, Something-Something, and Mini-Kinetics datasets, demonstrating strong accuracies at competitively-low flop counts.\n\nAdditional studies explore allowing the navigation module to make spatial, as well temporal, selections, and demonstrate that the learned navigation model achieves superior performance to other simpler sampling strategies.\n", "questions": "Why is TSN used as a baseline in Table 1? Although it's a great paper, the method is 6 years old.\n", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Originality** The main weakness is that the premise has already been explored thoroughly in other publications. Specifically in \"SlowFast Networks for Video Recognition\" https://arxiv.org/abs/1812.03982 (ICCV 2019, 1300+ citations). Both use the idea of a high frame rate low spatial resolution / channels stream, combined with a low frame rate high spatial resolution, with lateral connections from the fast pathway to the slow pathway. Both are targeted at limiting flops while boosting accuracy. \n\nMore recently in \"Multiview Transformers for Video Recognition\" (CVPR 2022) which extends the slow-fast premise to 2+ streams using transformer backbones, instead of CNNs.\n\nThe paper and accompanying analysis is incomplete without a direct comparison to SlowFast. \n\nThe primary difference is the adaptive selection of frames by the navigation module, vs. SlowFast's fixed temporal sampling rates plus lateral connections. Given that, at a quick glance the numbers seem comparable between the two(*), additional comparison between the approaches is warranted.\n\n**Clarity:** The paper is clearly written and generally easy to follow. Section 3.2 is a minor exception: I was eager to see the implicit temporal modeling, but found this explanation hard to follow. Perhaps it would benefit from more prose and fewer equations?\n\nLine 293: Please elaborate on how to \"sample frames from a gaussian distribution\", which is not obvious since Gaussian variables are continuous, not discrete.\n\n**Quality:** Generally good, except for the omission of highly-cited related work.\n\n**Significance:** As mentioned, this paper's significance is diminished by its limited originality.\n\n\n===\n\n(*) The original SlowFast paper publishes numbers on different datasets than this work, but from what I can piece together, SlowFast seems better or comparable. If it's fair to compare numbers on MiniKinetics and Kinetics-400, SlowFast gets 74.2% top-1 at 28.6 GFLOPs (see SlowFast Table 2 (b)), while this paper reports 73.5% top-1 at 22.0 GFLOPs. On Something-Something v2, SlowFast-ResNet50 seems to get 61.7 (see \"Multiview Transformers...\" Table 2 (c)) which is comparable to this paper's 61.3/62.5.\n\nI recognize the limitations of this analysis, and would love to see a proper apples-to-apples comparison.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "x5Q0wcuusc", "reviewer": "Reviewer_VKUT", "summary": "This paper proposes a two-branch network for efficient video recognition. In particular, the Ample Branch takes densely sampled input frames and processes them with reduced channel sizes. On the other hand, the Focal Branch only processes salient frames selected by a navigation module. Extensive experiments on multiple video benchmarks show that the proposed AFNet achieves state-of-the-art results with lower computational cost.", "questions": "1. The GFLOPs of RT=0.8 is a bit counterintuitive. Considering (1) the introduction of the addition Amber Branch, navigation module and (2) the last stage of the network remains unchanged, the GFLOPs of the model with RT=0.8 should be larger than 80% of that of the backbone model. This trend can be observed for the setting RT=0.4. However, for RT=0.8, GFLOPs = 31.7 is only 64.6% of TSM (GFLOPs=49.1), which is just slightly larger than RT=0.4 (GFLOPs=27.8). Please clarify this in the rebuttal.\n\n2. For the navigation module, instead of learning a sampling strategy, what if we learn a linear function that transforms the original T frames to T' steps? In other words, we aim to learn a T x T' weight matrix that compute T' weighted averages of the T frames. It's differentiable and easy to optimized with the rest of the model, and I'm curious whether it'll give worse results than doing frame sampling.", "limitations": "Yes.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1. The proposed architecture, AFNet, is simple yet effective for efficient video action recognition. As shown in the experiment section, AFNet achieves even better results than its baseline with lower computational cost. The idea of leveraging more input frames for avoiding information loss and salient frame selection is well-motivated, and the encouraging results provided in this paper are potentially insightful for the research community.\n2. The paper is overall well organized and well written.\n\nWeakness\n1. The novelty of individual components of AFNet is limited. For example, the two-branch design with downsampled \"fast branch\" is explored in SlowFast network (without dynamic selection of salient frames though);  the navigation module along with the Gumbel softmax optimization technique is also used in prior work for dynamic selection [1]. However, I believe that the contribution of the overall architecture design is sufficient and the proposed method achieves good results.\n\n2. Since Gumbel softmax is used for dynamic sampling of frames, the computational cost cannot be reduced during training. Although we usually care more about the computational cost of a model at inference, the cost of training will become a bottleneck if (1) the model is too large to fit into GPU memory (e.g., using more frames than 12 frames used in the paper); (2) the model training takes too much time. I understand it's out of the scope of this paper, but I'd suggest the author to try other \"hard-sampling\" algorithms for the selection module, for example, the perturbed maximum method [2, 3].\n\n3. Because the proposed two-branch design is generic to different backbone models, it's always better to see more experimental results with stronger backbones (e.g., ResNet-101 or even Transformers) and using more frames (12 frames are still a small number for long videos such as those in ActivityNet).\n\n\n[1] Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., Hsieh, C.J.: Dynamicvit: Efficient vision transformers with dynamic token sparsification. In: NIPS (2021)\n[2] Berthet, Quentin, et al. \"Learning with differentiable pertubed optimizers.\" Advances in neural information processing systems 33 (2020): 9508-9519.\n[3] Cordonnier, Jean-Baptiste, et al. \"Differentiable patch selection for image recognition.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "w4uRoZaLRHb", "reviewer": "Reviewer_aYAw", "summary": "This paper presents an efficient framework called Ample and Focal Network (AFNet) for video recognition. This framework consists of two branches: the ample branch preserves all input features by lightweight computation; the focal branch extracts features only from selected frames to save cost. By fusing the features of these two branches, AFNet can keep its focus on the crucial information while requiring less computation. Experiments on five datasets demonstrate the superiority of AFNet compared to state-of-the-art methods. However, I have some concerns about this paper. My detailed comments are as follows.", "questions": "1. The proposed method uses several navigation modules in a recurrent manner. What did the model learn to select in different stages?\n2. In section 3.1, the logit $p_n^t$ for frame t is generated with Eq.(3). Then for $p_n={p_n^t}_{t=1}^T$, all $p_n^t$ are with the same values as they are generated with the same convolution weights $W_2$ and feature $\\tilde{v}_{y_n^a}$. More discussions are required.\n3. The second term in Eq.(16) is used to contain the ratio of the selected frame with the square of the difference between $r$ and $RT$. However, This introduces an additional hyper-parameter $RT$ and restricts the model from selecting fewer frames for lower computation cost.\n4. InTab.1, AFNet achieves higher accuracy with fewer frames, which is counterintuitive. The explanation given by the authors is only for why AFNet achieves higher accuracy than TSN, not for why the fewer frames are selected, the higher accuracy AFNet achieves. More clear explanations are required.\n5. Why is AFNet not compared to other efficient frameworks like MoViNet[3]? More explanations are needed.\n6. Some action recognition methods are missing, such as Two-Stream Network [1] and T-C3D [2].\n\n[1]\t“Convolutional Two-Stream Network Fusion for Video Action Recognition.” CVPR (2016)\n[2]\t“T-C3D: Temporal Convolutional 3D Network for Real-Time Action Recognition.” AAAI (2018).\n[3]\t\"Movinets: Mobile video networks for efficient video recognition.\" CVPR (2021)\n\n", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper presents an efficient framework called Ample and Focal Network for video recognition, which uses more frames but reduces computation.\n2. The authors use a two-branch framework, in which two branches are complementary to each other, to prevent information loss when selecting fewer frames.\n3. The authors propose a navigation module that can select informative frames to save computational cost and is compatible with spatial adaptive works.\n\nWeaknesses:\n1. My biggest concern lies in the technical contribution of this paper. This method uses two streams, one for dealing with frames with high spatial resolution but low temporal resolution. The other process frames with low spatial resolution and high temporal resolution. Such an idea seems similar compared with the Slow-Fast [a] network.\n2. As for the adaptive frame selection, Scsampler [b] also uses a tiny network to select frames and a larger network to do action recognition. I suggest adding more discussions between them.\n3. In the experiments, the authors use only 12 frames (compared with 8-frame settings in previous methods), which is not convincing enough to verify the efficiency of the proposed methos.\n\n[a] SlowFast Networks for Video Recognition. ICCV 2019.\n[b] Compressing Videos to One Clip With Single-Step Sampling. CVPR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "cK33T7udapT", "reviewer": "Reviewer_ChDo", "summary": "This paper mainly targets action recognition. The authors advocate the importance of utilizing as many frames from each video as possible, while preserving proper computational cost. To this end they propose a novel architecture with two branches, one of which handles the whole video snippet with lower resolution and samples informative frames for fine-grained inference in another branch.", "questions": "1. In fact the claim that information loss leads to inferior performance is kind of counterfactual. Many previous studies have shown that for simple actions even one frame is enough for inference. On the other hand, for those complicated actions, temporal redundancy still exists. If the authors want to show the merit of utilizing all frames, it may be a good choice to directly train a model with all frames regardless of computational cost to show if it can significantly outperform the existing methods.\n2. The authors mention that outputs from two branches are merged using ‘specially designed fusion strategy’ (L120). However, the fusion strategy as in Eq. 8 is simply a weighted average with learnable weight. I am afraid that the authors overclaim their contribution on this module.\n3. The authors utilize residual connection between layers in the focal branch. This is questionable since different layers inquires potentially different frames due to the designed navigation module, which means features from different layers are not aligned in temporal dimension. I wonder if it is proper to directly add these features together.\n4. The authors claim that the proposed method do not build temporal modeling module. I am not sure whether the $W_2$ in Eq. 2 plays such a role to interact among frames.\n", "limitations": "Several points of limitations are mentioned, which is comprehensive and provides promising future works for the current method.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The idea of end-to-end frame selection is interesting.\n2. The authors provide abundant experiment to verify the effectiveness of their method.\n\n\nWeakness:\n1. The description is not clear enough. For example, the authors mention that the motivation of their method is to avoid ‘the loss of information compared to other dynamic methods’. However, it is not shown in this paper what kind of information loss exists in previous methods, and how such loss does harm to the performance. It would be better if more pilot study can be provided.\n2. The proposed two-branch structure with different temporal scale is similar to SlowFast [1], which is one of the most famous action recognition models and also builds lateral connection between a slow branch and a fast branch with difference frame rates. The authors should consider discussing about this paper when introducing the proposed method.\n\n[1] Feichtenhofer C, Fan H, Malik J, et al. Slowfast networks for video recognition[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 6202-6211.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "BaOjV3hSrh0", "reviewer": "Reviewer_ztLC", "summary": "This paper proposes Ample and Focal Network (AFNet) for video recognition. Specifically, the network has an ample and a focal branch. The ample branch operates on a set of neighboring (with strides) frames with reduce-sized feature maps (in height, width and channels), whereas the focal networks then takes in both input frames and intermediate predictions from focal network to deceptively process selected frames, with higher computation budget. The resulting network is claimed to have a better accuracy-computation trade-off than previous work. The authors conducted experiments on Something-Something v1/v2, Mini-Kinetics, Jester and ActivityNet and demonstrated that their method yields better accuracy compared to several baselines (e.g. TSM, AdaFuse-TSM, bLVNet etc) on these datasets.    \n", "questions": "- L150, isn't L_n a continuous vector computed using Eq 6? Is there some thresholding used here before selecting non-zero values? \n- Eq 1, is it necessary to keep the v notation on the left hand side of the equation? Since it represents input video, keeping it here might cause some confusion. \n- L145, \"we let tau decrease from 1 to 0.01 during train\", what's the decay schedule used? Do different schedulings make a difference? \n- Table 1, the improvement from TSN to AFNet is quite significant, and honestly a bit surprising. Does the author investigate the possibility of overfitting for the TSN results? \n", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The two-branch design makes intuitive sense, with one focus on lightweight processing on dense inputs, while the other process sparsely selected inputs with heavier computation.\n+ The paper is comprehensive in structure --- in addition to the qualitative description of the approach, the authors also included a section on the theoretical analysis of implicit temporal modeling. \n+ The result section includes comparison to several baseline methods across several datasets. It seems the proposed method has an edge on accuracy-computation trade-off across these comparisons. There are also some qualitative visualizations and ablations to dissect the approach. \n\nWeaknesses\n- As shown in Figure 2, the frame selection process is implemented as sparse convolutions, for which from the texts I cannot tell how efficient they are. This becomes more of an issue since in all tables the authors report FLOPs rather than actual inference latency. \n- Across Sth-Sth (Table 2), Mini-Kinetics (Table 3), and ActivityNet (Table 4), the accuracy/computation gains over the competitor is definitely not to a level that I'll consider significant. \n- There is no promise on code release, which might make it hard to reproduce the reported results. \n- For related work, the proposed approach missed an important citation: SlowFast Networks from Feichtenhofer et al. As it also sits on this two-branch idea for video recognition where one lightweight branch focuses on motion and another heavy branch focuses on semantics. This should be added and discussed. Another related paper is \"End-to-end Learning of Action Detection from Frame Glimpses in Videos\" from Yeung et al., as it first proposed to selective focus on a subset of frames for video recognition. \n\nWriting/Typos\n- L58, \"but strength the representation\" --> \"but strengthen the representation\"\n- L125, C_o, H_o and W_o are not introduced anywhere up until this point. \n- L273, \"to analysis the results\" --> \"to analyze the results\"\n- In L136, t in p^t_n is used to index frames, whereas in Eq 4, the superscript is overloaded to denote the frame selection flag. This will cause some confusions. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept