Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a two-branch network for efficient video recognition. In particular, the Ample Branch takes densely sampled input frames and processes them with reduced channel sizes. On the other hand, the Focal Branch only processes salient frames selected by a navigation module. Extensive experiments on multiple video benchmarks show that the proposed AFNet achieves state-of-the-art results with lower computational cost.
strengths_and_weaknesses: Strength
1. The proposed architecture, AFNet, is simple yet effective for efficient video action recognition. As shown in the experiment section, AFNet achieves even better results than its baseline with lower computational cost. The idea of leveraging more input frames for avoiding information loss and salient frame selection is well-motivated, and the encouraging results provided in this paper are potentially insightful for the research community.
2. The paper is overall well organized and well written.

Weakness
1. The novelty of individual components of AFNet is limited. For example, the two-branch design with downsampled "fast branch" is explored in SlowFast network (without dynamic selection of salient frames though);  the navigation module along with the Gumbel softmax optimization technique is also used in prior work for dynamic selection [1]. However, I believe that the contribution of the overall architecture design is sufficient and the proposed method achieves good results.

2. Since Gumbel softmax is used for dynamic sampling of frames, the computational cost cannot be reduced during training. Although we usually care more about the computational cost of a model at inference, the cost of training will become a bottleneck if (1) the model is too large to fit into GPU memory (e.g., using more frames than 12 frames used in the paper); (2) the model training takes too much time. I understand it's out of the scope of this paper, but I'd suggest the author to try other "hard-sampling" algorithms for the selection module, for example, the perturbed maximum method [2, 3].

3. Because the proposed two-branch design is generic to different backbone models, it's always better to see more experimental results with stronger backbones (e.g., ResNet-101 or even Transformers) and using more frames (12 frames are still a small number for long videos such as those in ActivityNet).


[1] Rao, Y., Zhao, W., Liu, B., Lu, J., Zhou, J., Hsieh, C.J.: Dynamicvit: Efficient vision transformers with dynamic token sparsification. In: NIPS (2021)
[2] Berthet, Quentin, et al. "Learning with differentiable pertubed optimizers." Advances in neural information processing systems 33 (2020): 9508-9519.
[3] Cordonnier, Jean-Baptiste, et al. "Differentiable patch selection for image recognition." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
questions: 1. The GFLOPs of RT=0.8 is a bit counterintuitive. Considering (1) the introduction of the addition Amber Branch, navigation module and (2) the last stage of the network remains unchanged, the GFLOPs of the model with RT=0.8 should be larger than 80% of that of the backbone model. This trend can be observed for the setting RT=0.4. However, for RT=0.8, GFLOPs = 31.7 is only 64.6% of TSM (GFLOPs=49.1), which is just slightly larger than RT=0.4 (GFLOPs=27.8). Please clarify this in the rebuttal.

2. For the navigation module, instead of learning a sampling strategy, what if we learn a linear function that transforms the original T frames to T' steps? In other words, we aim to learn a T x T' weight matrix that compute T' weighted averages of the T frames. It's differentiable and easy to optimized with the rest of the model, and I'm curious whether it'll give worse results than doing frame sampling.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper mainly targets action recognition. The authors advocate the importance of utilizing as many frames from each video as possible, while preserving proper computational cost. To this end they propose a novel architecture with two branches, one of which handles the whole video snippet with lower resolution and samples informative frames for fine-grained inference in another branch.
strengths_and_weaknesses: Strength:
1. The idea of end-to-end frame selection is interesting.
2. The authors provide abundant experiment to verify the effectiveness of their method.


Weakness:
1. The description is not clear enough. For example, the authors mention that the motivation of their method is to avoid ‘the loss of information compared to other dynamic methods’. However, it is not shown in this paper what kind of information loss exists in previous methods, and how such loss does harm to the performance. It would be better if more pilot study can be provided.
2. The proposed two-branch structure with different temporal scale is similar to SlowFast [1], which is one of the most famous action recognition models and also builds lateral connection between a slow branch and a fast branch with difference frame rates. The authors should consider discussing about this paper when introducing the proposed method.

[1] Feichtenhofer C, Fan H, Malik J, et al. Slowfast networks for video recognition[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 6202-6211.
questions: 1. In fact the claim that information loss leads to inferior performance is kind of counterfactual. Many previous studies have shown that for simple actions even one frame is enough for inference. On the other hand, for those complicated actions, temporal redundancy still exists. If the authors want to show the merit of utilizing all frames, it may be a good choice to directly train a model with all frames regardless of computational cost to show if it can significantly outperform the existing methods.
2. The authors mention that outputs from two branches are merged using ‘specially designed fusion strategy’ (L120). However, the fusion strategy as in Eq. 8 is simply a weighted average with learnable weight. I am afraid that the authors overclaim their contribution on this module.
3. The authors utilize residual connection between layers in the focal branch. This is questionable since different layers inquires potentially different frames due to the designed navigation module, which means features from different layers are not aligned in temporal dimension. I wonder if it is proper to directly add these features together.
4. The authors claim that the proposed method do not build temporal modeling module. I am not sure whether the $W_2$ in Eq. 2 plays such a role to interact among frames.

limitations: Several points of limitations are mentioned, which is comprehensive and provides promising future works for the current method.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper presents an efficient framework called Ample and Focal Network (AFNet) for video recognition. This framework consists of two branches: the ample branch preserves all input features by lightweight computation; the focal branch extracts features only from selected frames to save cost. By fusing the features of these two branches, AFNet can keep its focus on the crucial information while requiring less computation. Experiments on five datasets demonstrate the superiority of AFNet compared to state-of-the-art methods. However, I have some concerns about this paper. My detailed comments are as follows.
strengths_and_weaknesses: Strengths:
1. This paper presents an efficient framework called Ample and Focal Network for video recognition, which uses more frames but reduces computation.
2. The authors use a two-branch framework, in which two branches are complementary to each other, to prevent information loss when selecting fewer frames.
3. The authors propose a navigation module that can select informative frames to save computational cost and is compatible with spatial adaptive works.

Weaknesses:
1. My biggest concern lies in the technical contribution of this paper. This method uses two streams, one for dealing with frames with high spatial resolution but low temporal resolution. The other process frames with low spatial resolution and high temporal resolution. Such an idea seems similar compared with the Slow-Fast [a] network.
2. As for the adaptive frame selection, Scsampler [b] also uses a tiny network to select frames and a larger network to do action recognition. I suggest adding more discussions between them.
3. In the experiments, the authors use only 12 frames (compared with 8-frame settings in previous methods), which is not convincing enough to verify the efficiency of the proposed methos.

[a] SlowFast Networks for Video Recognition. ICCV 2019.
[b] Compressing Videos to One Clip With Single-Step Sampling. CVPR 2022.
questions: 1. The proposed method uses several navigation modules in a recurrent manner. What did the model learn to select in different stages?
2. In section 3.1, the logit $p_n^t$ for frame t is generated with Eq.(3). Then for $p_n={p_n^t}_{t=1}^T$, all $p_n^t$ are with the same values as they are generated with the same convolution weights $W_2$ and feature $\tilde{v}_{y_n^a}$. More discussions are required.
3. The second term in Eq.(16) is used to contain the ratio of the selected frame with the square of the difference between $r$ and $RT$. However, This introduces an additional hyper-parameter $RT$ and restricts the model from selecting fewer frames for lower computation cost.
4. InTab.1, AFNet achieves higher accuracy with fewer frames, which is counterintuitive. The explanation given by the authors is only for why AFNet achieves higher accuracy than TSN, not for why the fewer frames are selected, the higher accuracy AFNet achieves. More clear explanations are required.
5. Why is AFNet not compared to other efficient frameworks like MoViNet[3]? More explanations are needed.
6. Some action recognition methods are missing, such as Two-Stream Network [1] and T-C3D [2].

[1]	“Convolutional Two-Stream Network Fusion for Video Action Recognition.” CVPR (2016)
[2]	“T-C3D: Temporal Convolutional 3D Network for Real-Time Action Recognition.” AAAI (2018).
[3]	"Movinets: Mobile video networks for efficient video recognition." CVPR (2021)


limitations: The authors adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: This paper proposes Ample and Focal Network (AFNet) for video recognition. Specifically, the network has an ample and a focal branch. The ample branch operates on a set of neighboring (with strides) frames with reduce-sized feature maps (in height, width and channels), whereas the focal networks then takes in both input frames and intermediate predictions from focal network to deceptively process selected frames, with higher computation budget. The resulting network is claimed to have a better accuracy-computation trade-off than previous work. The authors conducted experiments on Something-Something v1/v2, Mini-Kinetics, Jester and ActivityNet and demonstrated that their method yields better accuracy compared to several baselines (e.g. TSM, AdaFuse-TSM, bLVNet etc) on these datasets.    

strengths_and_weaknesses: Strengths
+ The two-branch design makes intuitive sense, with one focus on lightweight processing on dense inputs, while the other process sparsely selected inputs with heavier computation.
+ The paper is comprehensive in structure --- in addition to the qualitative description of the approach, the authors also included a section on the theoretical analysis of implicit temporal modeling. 
+ The result section includes comparison to several baseline methods across several datasets. It seems the proposed method has an edge on accuracy-computation trade-off across these comparisons. There are also some qualitative visualizations and ablations to dissect the approach. 

Weaknesses
- As shown in Figure 2, the frame selection process is implemented as sparse convolutions, for which from the texts I cannot tell how efficient they are. This becomes more of an issue since in all tables the authors report FLOPs rather than actual inference latency. 
- Across Sth-Sth (Table 2), Mini-Kinetics (Table 3), and ActivityNet (Table 4), the accuracy/computation gains over the competitor is definitely not to a level that I'll consider significant. 
- There is no promise on code release, which might make it hard to reproduce the reported results. 
- For related work, the proposed approach missed an important citation: SlowFast Networks from Feichtenhofer et al. As it also sits on this two-branch idea for video recognition where one lightweight branch focuses on motion and another heavy branch focuses on semantics. This should be added and discussed. Another related paper is "End-to-end Learning of Action Detection from Frame Glimpses in Videos" from Yeung et al., as it first proposed to selective focus on a subset of frames for video recognition. 

Writing/Typos
- L58, "but strength the representation" --> "but strengthen the representation"
- L125, C_o, H_o and W_o are not introduced anywhere up until this point. 
- L273, "to analysis the results" --> "to analyze the results"
- In L136, t in p^t_n is used to index frames, whereas in Eq 4, the superscript is overloaded to denote the frame selection flag. This will cause some confusions. 

questions: - L150, isn't L_n a continuous vector computed using Eq 6? Is there some thresholding used here before selecting non-zero values? 
- Eq 1, is it necessary to keep the v notation on the left hand side of the equation? Since it represents input video, keeping it here might cause some confusion. 
- L145, "we let tau decrease from 1 to 0.01 during train", what's the decay schedule used? Do different schedulings make a difference? 
- Table 1, the improvement from TSN to AFNet is quite significant, and honestly a bit surprising. Does the author investigate the possibility of overfitting for the TSN results? 

limitations: Yes
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposed to reduce the cost and boost the accuracy of CNN models applied to video action recognition via a two-stream approach. The first, "ample" stream processes all of the frames, but cheaply, by using low spatial resolution and number of channels. The second "focal" stream processes frames at high-resolution, but only processes a few of the input frames. A navigation model uses the input from the ample stream to select which frames the focal stream should process, using Gumbel softmax.

The paper presents results on the ActivityNet, Something-Something, and Mini-Kinetics datasets, demonstrating strong accuracies at competitively-low flop counts.

Additional studies explore allowing the navigation module to make spatial, as well temporal, selections, and demonstrate that the learned navigation model achieves superior performance to other simpler sampling strategies.

strengths_and_weaknesses: **Originality** The main weakness is that the premise has already been explored thoroughly in other publications. Specifically in "SlowFast Networks for Video Recognition" https://arxiv.org/abs/1812.03982 (ICCV 2019, 1300+ citations). Both use the idea of a high frame rate low spatial resolution / channels stream, combined with a low frame rate high spatial resolution, with lateral connections from the fast pathway to the slow pathway. Both are targeted at limiting flops while boosting accuracy. 

More recently in "Multiview Transformers for Video Recognition" (CVPR 2022) which extends the slow-fast premise to 2+ streams using transformer backbones, instead of CNNs.

The paper and accompanying analysis is incomplete without a direct comparison to SlowFast. 

The primary difference is the adaptive selection of frames by the navigation module, vs. SlowFast's fixed temporal sampling rates plus lateral connections. Given that, at a quick glance the numbers seem comparable between the two(*), additional comparison between the approaches is warranted.

**Clarity:** The paper is clearly written and generally easy to follow. Section 3.2 is a minor exception: I was eager to see the implicit temporal modeling, but found this explanation hard to follow. Perhaps it would benefit from more prose and fewer equations?

Line 293: Please elaborate on how to "sample frames from a gaussian distribution", which is not obvious since Gaussian variables are continuous, not discrete.

**Quality:** Generally good, except for the omission of highly-cited related work.

**Significance:** As mentioned, this paper's significance is diminished by its limited originality.


===

(*) The original SlowFast paper publishes numbers on different datasets than this work, but from what I can piece together, SlowFast seems better or comparable. If it's fair to compare numbers on MiniKinetics and Kinetics-400, SlowFast gets 74.2% top-1 at 28.6 GFLOPs (see SlowFast Table 2 (b)), while this paper reports 73.5% top-1 at 22.0 GFLOPs. On Something-Something v2, SlowFast-ResNet50 seems to get 61.7 (see "Multiview Transformers..." Table 2 (c)) which is comparable to this paper's 61.3/62.5.

I recognize the limitations of this analysis, and would love to see a proper apples-to-apples comparison.

questions: Why is TSN used as a baseline in Table 1? Although it's a great paper, the method is 6 years old.

limitations: yes
review_title: null

Meta-review: metareview: The proposed architecture, AFNet, is simple yet effective for end-to-end efficient video action recognition.
It has good idea, well written paper, and relative solid experimental results to support the claim. 
The emergency reviewer gives the highest score (6), while the other reviewers do have some concerns with this paper.
Most of the other reviewers give the rating of borderline accept, after the authours make more efforts in the rebuttal period. In term of these, the initial recommendation would be acceptance.
confidence: Less certain
Decision: Accept