Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary
The paper presents a framework to reduce internal redundancy in the video recognition model. To do so, given the input frames, the framework predicts two scaling factors to conduct temporal and channel dimension reduction. The remaining part is reconstructed by cheap operations. The authors show that the framework achieves favorable results on several benchmarks.

Strengths
+ The paper is well written.
+ Strong quantitative and qualitative results (Consistent improvement over the state-of-the-art baselines).
+ Solid ablation studies and analysis.

Weakness
- The core idea of using a light-weight neural module to maximize the video model efficiency is not novel.
For example, AR-Net already has shown that a policy network can decide the video input resolution, i.e., spatial dimension, adaptively, leading to improve both efficiency and accuracy. In this work, the key concept is basically the same and only the primitive operation unit, i.e., temporal and channel dimensions, is different.

- Please provide experiment-level comparisons with AR-Net

- The following SOTA video models are missing in the reference.
DynamoNet: Dynamic action and motion network, ICCV 2019
Video Modeling with Correlation Networks, CVPR 2020
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition, ECCV2020
Directional Temporal Modeling for Action Recognition, ECCV2020

Question

- Can the idea be applied to dense video understanding tasks, such as video semantic segmentation?

Rating

- Aiming at acquiring an efficient model in a data-driven manner is indeed important for video models.
While the key idea is not novel, the paper has obvious empirical contributions that may help communities to invigorate future researches in this direction. 

review_title: Review for VA-RED: Video Adaptive Redundancy Reduction

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:
1. The motivation of the submission is clear and the writing is easy to follow. The problem being addressed is a practical and important issue in deploying video models to real applications. 

2. Experiments are thorough. This submission has done experiments on both action recognition and action localization task. The performance is promising on most video networks. 

Cons:
1. This submission proposes to address the problem of slow video inference, but the only metric they report is the Gflops. We know that Gflops is not a good indicator for speed comparison. Sometimes, a small Gflops does not mean fast speed. So I recommend to add a "speed" column in every table of this submission, to compare the actual inference speed among the methods. 

2. Regarding the searching method, can authors justify what are the differences to X3D? X3D also search the optimal combination of temporal dimension and channel dimension. Are the differences only on adding efficiency loss?

3. The efficiency loss is widely used in searching efficient image models, like in Proxylessnas. What is the contribution and motivation of using it here in searching video models? 
review_title: This submission proposes to adaptively reduce redundancy in temporal and channel dimension to make existing video networks more efficient.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a novel framework called $\text{VA-RED}^2$ to reduce spatial and temporal features to be computed for video understanding, which can reduce FLOPs when inferencing the video but remains the performance. 

The authors have done extensive experiments on video action recognition tasks and spatio-temporal action localization task in the area of video understanding. For the video action recognition task, experiments are carried out using Mini-Kinetics-200, Kinetics-400, and Moments-In-time datasets. For the action localization task, J-HMDB-21 dataset is used. Results show that this framework is promising, which reduces the computation but main the performance. 


Question: 
1. X3D-M in the original paper achieved at top-1 74.6 for Kinetics-400 dataset (but in table 4 reports clip-1: 61.8, video-1: 67.9) and FLOPs is 4.73 (6.20 reported in the paper). Can authors explain why there is a difference here? 


Minor:
1. Reduce ratio fact for channel-wise dynamic convolution $r=\frac{1}{2}^{p_c}$ such as in Fig.2 on page 4 and equation 7, equation 8 in supplementary materials on page 13. I think it would make more sense representing it as $(\frac{1}{2})^{p_c}$. 


review_title: This paper proposes a novel framework called $\text{VA-RED}^2$ to reduce spatial and temporal features to be computed for video understanding, which can reduce FLOPs when inferencing the video but remains the performance.  This paper is well-written and conducts extensive experiments to validate the performance of proposed approach.

Meta-review: This paper presents work on efficient video analysis.  The reviewers appreciated the clear formulation and effective methodology.  Concerns were raised over empirical validation.  The authors' responses added additional material that assisted in clarifying these points.  After the discussion the reviewers converged on a unanimous accept rating.  The paper contains solid advances in efficient inference for video analysis and is ready for publication.
Decision: Accept (Poster)