Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: - The overall quality is good. It addresses the training issue of 3D convolutional network in a novel perspective and shows promising results. The presentation is clear except for some grammar mistakes. 

Detailed Comments:

- In Sec. 3.2, how is the transition performance evaluated is not clear. What is the cost / gain from i to j? Is it the same for all states?

- Ablation experiments should be provided to show the performance difference using different paths. Is the adopted path actually the optimal path? And how about if some intermediate states are skipped?

- The proposed optimization planning doesn't seem to work specifically for the 3D convnets and for the action recognition task. As long as we need to train a network in stages with multiple hyper-parameters, this should apply theoretically. 

- Abstract: 'follows' --> 'is followed'
review_title: This paper proposes an optimization planning strategy to systematically train a 3D network. It shows promising results on multiple datasets for the tasks of action recognition. 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Overview:
The paper proposes a novel way to automatically tune 3D ConvNet hyper-parameters (learning rate, input clip length, sampling way). This is achieved by decomposing the optimization path into several states and the state transition is triggered when the knee-point on the performance-epoch curve is met. Extensive experiments are conducted on popular video benchmarks and show that the optimization planning is effective to improve the accuracy and requires less time time compared to the hand-tuned procedure.


Strengths:

++ The paper provides a novel perspective of designing the hyper-parameters for 3D ConvNets. The automatic planning through the optimization path alleviate the tedious work of hand tuning and save a lot of times: Once the knee-point on the performance-epoch curve is achieved, we can easily transit to the next optimization state (with another set of hyper-parameters).


++ The experiments are thorough. The authors conduct experiments on all popular video recognition benchmarks. Also the experiment results corroborate the effectiveness of optimization planning. 

Weaknesses:

-- The construction of transition graph has some strict principles that could be relaxed. 
  (1) For scheduling learning rate, It only considers that "training starts from high learning rate". However, previous works (e.g. ICLR2017: https://arxiv.org/abs/1608.03983) have shown that cyclic cosine learning schedule might benefit training. Therefore, I don't see the necessity that we put this restriction.
  (2) For scheduling input length, this is not the first time that people do this. For example, Wang et. al. (CVPR2018, https://arxiv.org/abs/1711.07971) first train using 64 frames as input and then finetune on 128 frames. Also, Wu et. al. (CVPR2020, https://arxiv.org/abs/1912.00998) proposes another way of adjust the sampling strategy.


-- The comparison is mostly conducted on the newly proposed DG-P3D family architectures. It would be better if there is an additional row of results on the standard I3D/SlowFast network in Table 3. (this is somewhat minor and not required considering the time limit)


-- "82.5%" (with flow stream) is not the "new record" for Kinetics-400. ir-CSN-152 (ICCV2019, https://arxiv.org/abs/1904.02811) can achieve 82.6% on Kinetics-400 using **RGB only**. OmniSource (ECCV2020, https://arxiv.org/abs/2003.13042) improves it to 83.6% using additional data.




I have some additional questions:

-- In Figure 4, it shows that SS-V1/2 would prefer uniform sampling however Kinetics would prefer consecutive sampling. However, this seems somewhat counter-intuitive to me. We know that Kinetics have videos with static scenes or slow motion while SS-V1/2 have some videos with more intensive motion. Does it make more sense to use dense frames (consecutive) to recognize intensive motion?
review_title: Reviewer #2

final_score: rating: 6: Marginally above acceptance threshold
review_content: - The paper proposed a Dual-head Global-contextual Pseudo-3D (DG-P3D) network and an automated optimization path to train 3D ConvNet for action recognition. The proposed method is evaluated on various of action recognition datasets, and achieved convincing results.

- Another contribution of the paper is its decreased cost. It mentioned that the time cost for grid optimization planning with 8 NVidia Titan V GPUs reduced the running time from 4057h to 288h on Kinetics.

- The optimization method is proposed for 3D ConvNet, However, the authors only benchmarked it with the new proposed DG-P3D network. It could be interesting to know what is the performance applying it to other basic 3D ConvNets. It is also interesting to know what is the performance of DG-P3D without this new optimization strategy.


review_title: Interesting paper about automate the optimization process of 3D ConvNets

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:
  This paper proposed two things:  
    1. hyper-parameter planning for training action recognition models.   
    2. a new 3D net architecture for action recognition. 

  The results show with the planning, authors can reduce training time significantly. and the proposed DG-P3D also is good for action recognition


Strong points:

  1. This paper explores a new approach for hyperparameter tuning to get a better model quickly.

  2. This paper performs extensive experiments to validate their proposed methods.

Weak points:

  1. It misses one very important reference on the same topic, "Multigrid Method for Efficiently Training Video Models" CVPR 2020. This paper also proposes a new way to efficiently train action recognition models. 

  2. The way authors compared to others are mixed, making that it is a little bit difficult to compare the contribution of the paper. 
  
    2.1 To be more fair compare to SOTA approaches in Table 4 and Table 5 with respect to the contributions of DG-P3D, the author should include the results without planning approach and use a similar training setting to the other SOTA approaches. Or apply the proposed optimization planning on those SOTA approaches to see the improvement. In this case, the advantages of each component will be more clear. 

    2.2 When comparing to other approaches, authors should either include the number of frames used or computational load (like FLOPs) into the consideration, instead of performance only. E.g., for Kinetics400, slowfast-ResNet-50 (77.0% top-1 acc) only use 32 frames while the proposed models for Kinetics400 use 128 frames (If I understand correctly); thus, simply comparing the accuracy is unfair. On the other hand, If authors want to compare the performance only, authors might want to include the model with different backbones, e.g. bLVNet-TAM has models achieved 53.1% on SS-V1 and 65.2% on SS-V2 with RGB only.


Questions:

  1. If a node has multiple destinations, e.g. node S5 in Figure 1a, how do the author select S7 or S8? In my understanding, the proposed algorithm will train the model at S5 with the parameters at S7 and S8 independently and then select the better one. Is it correct?

  2. Followed by 1, when encountering multiple destinations, if all destination nodes need to be explored before making the transition, do the author takes those time into account as well in Table 4? I do not find those in Figure 4, either.

  3. What is the delay parameter T in practical? Is it a dataset-dependent parameter?

  4. What is the inference protocol used in the evaluation? The authors mentioned there are two common approaches but do not mention which one is used.

Others:

 1. When comparing to SOTA, I think authors miss "X3D: Expanding Architectures for Efficient Video Recognition" CVPR 2020.

After rebuttal:

The authors addressed some of my concerns, and the proposed optimization plan is interested.

1. I still think that the proposed architecture does not outperform SOTA architecture, like SlowFast. 
  - First, in the rebuttal, the authors mention that with hand-crafted strategies, the proposed model is 0.4% better than SlowFast; nonetheless, the hand-crafted strategies train longer epochs than SlowFast. (SlowFast is trained with 196 epochs.) As authors propose new strategies, they know better how different strategies could affect. 
  - Second, in the rebuttal, the authors improve I3D by 1.7% with optimization planning, which means, SlowFast might outperform the proposed model if involving optimization planning. Moreover, in the revised Table 7, the SlowFast does not have more FLOPs than the proposed network under the same backbone. Simply checking Table 8 might be confused. 

2. About the optimization planning, it is still confused about how many epochs spent on the explored nodes. E.g. in Figure 4, for the Kinetics dataset, I do not understand why there is a black edge between S4 and S5 as S4 is not reached. (Authors noted that the black edges mean the explored strategies.) And why not there is no number on those explored edges? Another example, if we look at Table 3 and Figure 4 together, and again for the Kinetics dataset, the summation of those red numbers is 248, but this number is larger than any number in Table 3. As the authors mention that they included the epochs of the explored strategies in Table 3, that means they only spend 29 epochs in the exploration (if we treat Figure 4 for DG-P3D in Table 3.). For me, it seems too good to be true.

Thus, I keep my rating. 


review_title: Reviews

Meta-review: The reviewers appreciate the idea of hyperparameter planning and the thorough experimentation. Some concerns remain regarding the comparison between this method and SlowFast that require to be addressed. Also, the scope of the paper that targets hyperparameter optimization networks for action recognition specifically, may be too narrow for an ICLR audience. 
Decision: Reject