Optimization Planning for 3D ConvNets
Authors: Zhaofan Qiu
Ting Yao
Chong-wah Ngo
Tao Mei
Venue:  (2021)
Keywords: 3D ConvNets
Network Training
Video Recognition

Abstract: 3D Convolutional Neural Networks (3D ConvNets) have been regarded as a powerful class of models for video recognition. Nevertheless, it is not trivial to optimally learn a 3D ConvNets due to high complexity and various options of the training scheme. The most common hand-tuning process starts from learning 3D ConvNets using short video clips and then is followed by learning long-term temporal dependency using lengthy clips, while gradually decaying the learning rate from high to low as training progresses. The fact that such process comes along with several heuristic settings motivates the study to seek an optimal ``path'' to automate the entire training. In this paper, we decompose the path into a series of training ``states'' and specify the hyper-parameters, e.g., learning rate and the length of input clips, in each state. The estimation of the knee point on the performance-epoch curve triggers the transition from one state to another. We perform dynamic programming over all the candidate states to plan the optimal permutation of states, i.e., optimization path. Furthermore, we devise a new 3D ConvNets with a unique design of dual-head classifier to improve the spatial and temporal discrimination. Extensive experiments conducted on seven public video recognition benchmarks demonstrate the advantages of our proposal. With the optimization planning, our 3D ConvNets achieves superior results when comparing to the state-of-the-art video recognition approaches. More remarkably, we obtain the top-1 accuracy of 82.5% and 84.3% on the large-scale Kinetics-400 and Kinetics-600 datasets, respectively.

Under review as a conference paper at ICLR 2021
OPTIMIZATION PLANNING FOR 3D CONVNETS
Anonymous authors
Paper under double-blind review
ABSTRACT
3D Convolutional Neural Networks (3D ConvNets) have been regarded as a pow- erful class of models for video recognition. Nevertheless, it is not trivial to op- timally learn 3D ConvNets due to high complexity and various options of the training scheme. The most common hand-tuning process starts from learning 3D ConvNets using short video clips and then is followed by learning long-term tem- poral dependency using lengthy clips, while gradually decaying the learning rate from high to low training progresses. The fact that such process comes along with several heuristic settings motivates the study to seek an optimal "path" to automate the entire training. In this paper, decompose the path into se- ries of training "states" and specify the hyper-parameterss c.g., learning rate and the length of input clips, in each state. The estimation of the knee point on the performance-epech curve triggers the transition from one state to another. We perform dynamic programming over all the candidate states to plan the optimal permutation of states, i.e., optimization path. Furthermore, we devise a new 3D ConvNets with a unique design of dual-head classifier improve the spatial and temporal discrimination. Extensive experiments conducted on seven public video recognition benchmarks demonstrate the advantages of our proposal. With the op- timization planning, our 3D ConvNets achieves superior results when comparing to the state-of-the-art video recognition approaches. More remarkably, we ob- tain the top-1 accuracy of 82.5% and 84.3% on the large-scale Kinetics-400 and Kinetics-600 datasets, respectively.
INTRODUCTION
The recent advances in 3D Convolutional Neural Networks (3D ConvNets) have successfully pushed the limits and improved the state-of-the- of video recognition. For instance, an ensemble of LGD- 3D networks (Qiu et al., 2019) achieves 17.88 77.88 in terms of average error in trimmed video clas- sification task of ActivityNet Challenge 2019, which is dramatically lower than the error (29.3%) attained by the former 13D networks (Carreira & Zisserman 2017). The result basically indicates the advantage and great potential of 3D ConvNets for improving the performance of video recog- nition. Despite these impressive progresses, learning effective 3D ConvNets for video recognition remains challenging, due to large variations and complexities of video content. Existing works on 3D ConvNets (Tran et al. 2015 Carreira & Zisserman, 2017 Tran et al. 20188] Wang et al. 2018c) Feichtenhofer al. 2019 Qiu et al. 2017 2019) predominately focus on the designs of network architectures but seldom explore how to train a 3D ConvNets in a principled way.
The difficulty in training 3D ConvNets originates from the high flexibility of the training scheme. Compared to the training of 2D ConvNets (Ge et al., 2019 Lang et al., 2019; Yaida 2019). the involvement of temporal dimension in 3D ConvNets brings two new problems how many frames should be sampled from the video and how to sample these frames. First, the length of video clip tradeoff control the balance between training efficiency and long-range temporal modeling for learning 3D ConvNets. On one hand, training with short clips (16 frames) (Tran et al. 2015 Qiu et al. 2017) generally leads to fast convergence with large mini-batch, and also alleviates the overfitting problem through data augmentation brought by sampling short clips. On the other hand, recent works (Varol et al.) 2018 Wang et al. 2018c Qiu et al. 2019) have proven better ability in capturing long-range dependency when training with long clips (over 100 frames) at the expense of training time. The second issue is the sampling strategy. Uniform sampling (Fan et al.) (201 Jiang
et al. 2019 Martínez et al. 2019) offers the network fast-forward overview of the entire video,

---

Under review as a conference paper at ICLR 2021
while consecutive sampling (Tran et al. 2013: Qiu et al. 2017 2019 Varol et al. 2018 Wang al. 2018c) can better capture the spatio-temporal relation across frames. Given these complex choices of training scheme, learning powerful 3D ConvNets often requires significant engineering efforts of human experts to determine the optimal strategy on each dataset. That motivates us to automate the design of training strategy for 3D ConvNets.
In the paper, we propose optimization planning mechanism which seeks the optimal training strategy of 3D ConvNets adaptively. To this end, our optimization planning studies three problems: 1) choose between consecutive or uniform sampling; 2) when to increase the length of inpul clip: 3) when to decrease the learning rate. Specifically, we decompose the training process into several training states. Each state is assigned with the fixed hyper-parameters. including sampling strategy, length of input clip and learning rate. The transition between states represents the change of haper-parameeers during training. Therefore, the training process can be decided by the permutation of different states and the number of epochs for each state. Here, we build a candidate transition graph to define the valid transitions between states. The search of the best optimization strategy is then equivalent to seeking the optimal path from the initial state to the final state on the graph, which can be solved by dynamic programming algorithm. In order to determine the best epoch for each state in such process, we propose a knee point estimation method via fitting the performance-epode curve. In general, our optimization planning is viewed as training scheme controller and is readily applicable to train other neural networks in stages with multiple hyper-parameters.
To the best of our knowledge, our work is the first to address the issue of optimization planning for 3D ConvNets training. The issue also leads to the elegant view of how the order and epochs for different hyper-parameters should be planned adaptively. We uniquely formulate the problem as seeking an optimal training path and devise a new 3D ConvNets with dual-head classifier. Extensive experiments on seven datasets demonstrate the effectiveness of our proposal, and with optimization planning, our 3D ConvNets achieves superior results than several state-of-the the-thh-rt techniques.
2 RELATED WORK
The early works using Convolutional Neural Networks for video recognition are mostly extended from 2D ConyNets for image classification (Karpathy et al. [2014] Simonyan & Zisserman 2014] Feichtenhofer et al. 2016; Wang et al. 2016). These approaches often treat a video as a sequence of frames or optical flow images, and the pixel-level temporal evolution across consecutive frames are seldom explored. To alleviate this issue, 3D ConvNets in Ji et al. (2013) is devised to directly learn spatio-temporal representation from a short video clip via 3D convolution. Tran al. design widely-adopted 3D ConvNets in Tran et al.] (2015), namely C3D, consisting of 3D convolutions and 3D poolings optimized on the large-scale Sports (Karpathy et al. 2014) dataset. Despite having encouraging performances, the training of 3D ConyNets is computationally expensive and the model size suffers from a massive growth. Later in Qiu et al. (2017); ; Tran et al. (2018); Xic et al. (2018), the decomposed 3D convolution is proposed to simulate one 3D convolution with one 2D spatial convolution plus one 1D) temporal convolution. Recently, more advanced techniques are presented for 3D ConvNets, including inflating 2D convolutions (Carreira & Zisserman 2017). non-local pooling (Wang et al. 2018c) and docal-and-globe diffusion (Qiu et al. 2019).
Our work expands the research horizons of 3D ConvNets and focuses on improving 3D ConvNets training by adaptively planning the optimization process. The related works for 2D ConvNets train- ing (Chee & Toulis 2018 Lang et al. 2019 Yaida 2019) automate the training strategy via only changing the learning rate adaptively. Our problem is much more challenging especially when tem- poral dimension is additionally considered and involved in the training scheme of 3D ConvNets. For enhancing 3D ConvNets training, the recent works (Wang et al. 2018c| Qiu et al., 2019) first train 3D ConvNets with short input clips and then fine-tune the network with lengthy clips, which balances training efficiency and long-range temporal modeling. The multigrid method (Wu et al.) 2020) further cyclically changes spatial resolution and temporal duration of input clips for a more efficient optimization of 3D ConvNets. The research in this paper contributes by studying not only training 3D ConvNets with multiple lengths of input clips, but also adaptively scheduling the change of input clip length through optimization planning.
2

---

Under review as a conference paper at ICLR 2021
Donnec Sarsding IeS
Undierr Sempiinn Je!
Consestine Sumpling DOL
Lietoore Surnshing Soa
Tarm
Fomm
From
Fromer
(a) Basic transition graph
(b) Extended transition graph
Figure I: Examples of two transition graphs. The circles denote candidate states and the arrows represent candidate transitions. The ultimate model is the one with higher accuracy of the two final states.
OPTIMIZATION PLANNING
3.1 PROBLEM FORMULATION
The goal of optimization planning is to automate the learning strategy of 3D ConvNets. For- mally, the optimization process of 3D ConvNets can be represented as an optimization path P (So, 51, .... SN). which consists of one initial state Su and N intermediate states. Each in- termediate state assigned with the fixed hyper-parameters, and the training is performed with these N different settings
by one. The training epoch each setting is decided by T (4,,42...,144 (N),,,,,, J. in which I denotes the number of epochs when moving from S;-1 to S,. The hyper- parameters include sampling sttlalegg
(2,,2,,-.. IN, } and learning-rate {r1, T2... TN.h. TN. where cs and us denotes consecutive sampling and uniform sampling, respectively. In this case, there are N₁ x Nr valid types of training states.
{es, us), length-of iinpul_ length-of_inuul
The objective function of optimization planning is to seek the optimal strategy {P, T by maximiz- ing the performance of the final state Sx:
marimize V(SN),
where V(-) is the target performance, i.e., mean accuracy on validation set in our case.
3.2 OPTIMIZATION PATH
To plan the optimal permutation of training states, we first choose final state SN, which is usually with low learning rate and lengthy input clip. Then, the problem of seeking an optimal optimiza- tion path to SN is naturally decomposed to the subproblem of finding the optimization path to an intermediate state S; and the state transition from S; to SN As such, the problem can be solved by dynamic programming. Specifically, the solution of optimization path P(5x) can be given in a recursive form:
P(Sx) (P():-), 1" = argmaa (V(S, + Sx))
(2)
When executing the transfer from the state S. to the state SN. we fine-tune the 3D ConvNets at the state S, by using the hyper-parameters at the state Sy. We then evaluate such fine-tuned model on the validation set to measure the priority of this transition, i.e., V(S: - Sw)- We choose the state 5%, which achieves the highest priority of transition to the state Sy. as the preceding state of Sy. In other words, the optimal path for Sy derives from the best-performing preceding state S.-. Here, we propose to pre-define all the valid transitions in a directed acyclic graph and determine the best optimization path of each state one by one in the topological order. Figure I(a) shows one example of the pre-defined transition graph. In the example, we set the number of candidate input clip lengths N, 3 and the number of candidate learning rates N, = 3. Hence, there are x 3 = 18 candidate states. Then, the possible transitions, i.e., the connections between states, are determined by the following principles:
(1) The transitions between states with different sampling strategies are forbidden. We choose Sg and Si8 as the final states for consecutive sampling and uniform sampling, respectively.
(2) The training only starts from high learning rate and short input clips.
(3) The intermediate state can be only transferred to a new state, where either the learning rate is decreased the length of input clip is increased in the new state.
Please note that, some very specific learning rate strategics, e.g., schedules with restart or warmup, show that increasing the learning rate properly may benefit training. Nevertheless, there is still no
3

---

Under review as a conference paper at ICLR 2021
Table 1: The comparisons of four fitting functions in terms of RMSE and R- R-Square.
Fitting Fuaction (.)
Constraints
RMSE
R-Square
power: cr + ozlt + 1)08 + aut antt
Orgy Cryy Ots
1,010 x 10
0.356
multi-power: du + (t 1))s asll 1)nh + out + att?
dra, Oigy (K4, Oiy, Dry
1.030 10
0.320
exponential: ofe +ast + ast?
022 (3d, 05 <2 02, 03,, Ory
1.007 10
0.360
multi-exponential.oo 0n + age"8" #age"o" aut + aay?
1.063 x 10"
0.350
decu on Mowttatteeeeee Set
Accurcy va
Munber of Epochs
Number of Epochs
Number Esocts
(a)
(b)
(c)
Figure 2: The examples of (a) the collected performance-epoce curves: (b) the fitting results for training model from scratch; (c) the fitting results for fine-tuning model.
clear principle of when to increase the learning rate, and thus it is very difficult to automate these schedules. In the works of adaptively changing the learning rate for 2D ConvNets training (Ge al.,
2019
Lang et al. [019] Yaida 2019), such cyclic schedules are also not taken into account. As result, we only consider the schedule of decreasing learning rate in the transition graph.
These principles can simplify the transition graph and reduce the time cost when solving Equ.(2) We take this graph as basic transition graph. Furthermore, we also build an extended transition graph by enabling simultaneously decreasing the input clip length and the learning rate, as shown Figure T(b) such graph, the training strategies are more flexible.
3.3 STATE TRANSITION
One state transition from S; to S, is defined as a training step that starts to optimize the model at S; by using the byper-parameter. at Sy. Then the question is when this training step completes. Here, we derive the spirit from SASA (Lang et al. 2019) that trains the network with constant hyper-parameters until it reaches a stationary condition. SASA presents to adaptively evaluate the convergence stochastic gradient descent by Yaida's ss condition (Yaida 2019) during training. How- ever, in practice, the thoroughly optimized network does not always perform well validation set due to overfitting problem. Therefore, we take both convergence and overfitting into account, and propose to estimate the knee point the performance-epoch curve evaluated on the validation set, which performs more steadily across various datasets. Specifically, we measure the accuracy y/2 by evaluating the intermediate model after t-th training epoch on validation set. To estimate the knee point given limited number of observations yt, we fit the curve by a continuous function fa (t) as
3/ (t) +
where 2n is the stochastic factor following normal distribution, and a denotes the parameters of function f. Here, we choose (t) as unimodal function ensure that there only one maximum value. The curve fitting can be formulated as the optimization of parameters by minimizing the distance between observed performance and estimated performance:
minimize El (+)1, s.f. fa( is unimodal.
(4)
We exploit Trust Region Reflective algorithm (Branch et al. 1999) to solve this problem and the algorithm is robust for arbitrary form of function (t). To adaptively stop the iteration, we estimate the knee point epoch t by solving Equ.(4) after each training epoch. If the current epoch is larger than t* T. will stop the iteration and choose t* the best epoch number. Here, T is delay parameter which allows the model to have a T -epoch attempt even if t > t". We simply fix the delay parameter T to 10 in all the experiments.
Next, the essential issue is the form of fitting function (t). We separate the function into two parts fa (1) gm (t) + h. (1), where ga (1) is increasing bounded function to simulate the convergence
4

---

Under review as a conference paper at ICLR 2021
Global-Coniestull Pueudo-3D Bock
Dual-Heed Clasifier
TemparalHeed
Spatial
Pool
Res5
300494ch77
1948
Temporal
Res2
Res3
ResA
102tabelaa4
3eto/177a11
Hoal
Figure 3:
overview of our proposed Dual-head Global-contextual Pseudo-3D (DG-P3D) network. Here, we take the 16-frame input as an example and the size of output feature map is also given for each layer.
of the model, and (t) is a concave function to model the influence of overfitting. Table 1 shows four examples of fitting function (t). In the four functions, fix ha (t) as quadratic function and choose power, multi-power, exponential and multi-exponentiall function as 9a (t), respec- tively. Please note that, for each function, some constraints are given guarantee the properties of (t) and ha (t). We empirically validate the functions by pre-collecting 162 performance-epoch curves (Figure 2(a) from the training processes of different networks on different datasets and em- ploying the four functions to fit the curves by solving Equ.(4). Table Tableeccomprrt compares the average Root Mean Square Error (RMSE) and R-square when using each function. Figure [(b) and Figure 2 further depict fitting example in the context of model training from scratch and model fine-tuning, respectively. The general observation is that, all the four functions can nicely fit the performance- epoch curve and do not make a major difference on the final performance. Thus, we simply choose the best- pers-rerfrring exponential function in the rest of the paper.
4 3D CONVNETS ARCHITECTURE
In this section, we present the proposed Dual-head Global-contextual Pseudo-3D (DG-P3D) net- work. An overview of the architecture is shown in Figure In particular, the network is originated from the residual network (He et al. 2016) and further extended to 3D manner with three designs, i... pseudo-3D convolution, global context and dual-head classifier.
Pseudo-3D convolution. To achieve good tradeoff between accuracy and computational cost, pseudo-3D convolution is proposed in Qiu et al. (2017) that decomposes 3D learning into 2D convo- lutions in spatial space plus ID operations in temporal dimension. The similar idea of decomposing 3D convolution is also presented in R(2+1)D (Tran et al.) 2018)) and S3D (Xic al. 2018). To sim- plify the decomposition, in this paper, we only choose P3D-A block with the highest performance in Qiu al. (017). which cascades the spatial convolution and temporal convolution in turn.
Global context. The recent works on lon-local networks (Wang et al. 2018c Cao et al. 2019 Qiu et al. 2019) highlight the drawback of performing convolutions, in which each operation processes only a local window of neighboring pixels and lacks a holistic view of field. To address this limi- tation, choose the simple way encapsulate global context that learns the global residual from the global-pooled representation and then broadcasts to each position in the feature map.
Dual-head classifier. 3D ConvNets are expected to have both spatial and temporal discrimination. For example, the SlowFast network (Feichtenhhoer et al., 2019) contains one separate pathway for visual appearance and temporal dynamics, respectively. Here, we uniquely propose a simpler way that builds a dual-head classifier at the top of the network instead of the two-path structure in the SlowFast network. between, the temporal head with large temporal dimension focuses on model- ing the temporal evolution, and the spatial head with large spatial resolution emphasizes the spatial discrimination. The predictions from two heads are linearly fused. As such, our design costs less computations and is easier implement.
EXPERIMENTS
5.1 DATASETS
The experiments are conducted on HMDB51, UCF101, ActivityNet, SS-VI/V2, Kineticy-400 and Kinetics-600 datasets. details the information and settings these datasets. The HMDB51
5

---

Under review as a conference paper at ICLR 2021
Table 2:
The number of videos, the number of target categories and the detailed settings for optimization
planning on HMDB51, UCF101, ActivityNet, SS-V1, V1, SS- SS-V2, Kinetics-400 and Kinetics-600 datasets.
Dataset
Avideos
#classes
12
Is
F1
F3
Dropout
HMDBS1
6K
51
16
32
64
0.01
0.001
0.0001
0.9
UCFI01
0.01
0.0001
NctivityNeet
20K
200
16
32
128
0.01
0.001
0.0001
0.9
SS-VI
108K
174
0.04
0.004
0.0004
0.5
SS-V2
220K
174
32
0.04
0.004
0.9004
0.5
Kinetics 400
300K
400
16
32
128
0.04
0.004
0.0004
0.5
Kinetica-600
0.04
0.0004
(Kuchne al. [201]. UCFI01 (Soomro al.) 2012), Kinetics-400 (Carreira & Zisserman 2017)
and Kinetics-600 (Carreira et al. 2018) are the most popular video benchmarks for action recog- Nomething-Something
nition on trimmed video clips. The Something- Something VI (SS-V1) dataset is firstly con- structed in Gooal et al. (217)) to learn fine-gre ane-ggriied human-object interactions, and then extended to Sooteiinn- V2 (SS-V2) recently. The ActivityNet (Caba Heilbron et al.) 2015) dataset is an untrimmed video benchmark for activity recognition. The latest released version of the dataset (v1.3) is exploited. In our experiments, we only use the video-level label of ActivityNet and disable the temporal annotations. Note that the labels for test sets not publicly available, and thus the performances of ActivityNet, SS-VI, V2. Kinetics-400 and Kinetics-66l are all reported on the validation set. For optimization planning, the original training set of each dataset is split into two parts for learning the network weights and validating the performance, respectively. We construct this internal validation set with the same size the original validation/test set. Note that the original validation/test set is never exploited in the optimization planning.
5.2 IMPLEMENTATION DETAILS
For optimization planning, we set the number of choices for both input clip length N, and learning rate N, as 3, and utilize the extended transition graph introduced in Section 3.2 The candidate values of input clip length {41 12, (3) and learning rate {ri, 72, } for each dataset are summarized in Table [ Specifically, on SS-VI, V2, Kinetics-400 and Kinetics-600 600 datasets, the base learning rate is set as 0.04 and the dropout ratio is fixed as 0.5. For HMDB51, UCF101 and ActivityNett we set lower base learning rate and higher dropout ratio due to limited training samples. The maximum clip length is 64 for HMDB51 and UCF101, while increased to 128 for ActivityNet, Kinetics-400 and Kinetics-600 Considering that the video clips SS-VI and SS-V2 are usually shorter than 64 frames, we only use two settings, i.c., 16-frame and 32-frame, for the input clip.
The network training in this paper is implemented on Caffe (Jia et al. 2014) framework and the mini-batch stochastic gradient descent is employed to tune the network. The resolution of the input clip is fixed as 224 x 224, which is randomly cropped from the video clip resized with the short size in [256,3] 340). The clip is randomly flipped along horizontal direction for data augmentation except for SS- V1 and SS- V2 view of the direction-related categories. Following the settings Wang et al. (2018c); | Qiu et al. (2019), for the network training with long clips (64-frame and 128 128-frame' frame), we freeze the parameters of all Batch Normalization layers except for the first one since the batch size is too small for batch normalization.
There are two inference strategies for the evaluations. The first one roughly predicts the video label 224 224 single center crop from the centric one clip resized with the short size 256. This strategy is only used when planning the optimization for the purpose of efficiency. Once the optimization path is fixed, train 3D ConvNets with the path and evaluate the learnt 3D ConvNets by using the second strategy, i.e., the three-crop strategy as in |eechhenhooerr et al. (2019), which crops three 256 x 256 regions from each video clip. The video-level prediction score is achieved by averaging all scores from 10 uniform sampled clips.
5.3 EVALUATION OF OPTIMIZATION PLANNING
We firstly verify the effectiveness of our proposed optimization planning for 3D ConvNets and com- pare the hand-tuner strategies. To find the most powerful hand-tuned strategy, capitalize on the popular practices in the literature, and grid-search the training settings through four dimensions, i.c., input length, learning decay, sampling strategy and training epochs. Specifically, for input clip

---

Under review as a conference paper at ICLR 2021
Table 3: The comparisons between optimization planning (OP) and hand-tuned strategies with different 3D ConvNets on Kinetics-400 dataset. The number in the bracket denotes the best number of epoches, which is achieved by grid-search for hand-tuncd tuned strategies and adaptively determined for our optimization planning.
Hand-tuned Strategies
Network
Sampling
I1 la
l
la
I1 + l2 - la
3-step
cosine
3-stop
cosine
3-step
cosine
P3D
consecutivee
74.5 (256)
74.4 (320)
75.4 (320)
75.3 (384)
(320)
75.2
76.8 (184)
uniform
(192)
74.3 (256)
74.7 (256)
(192)
74.9 (256)
75.2 (192)
G-P3D
consecutive
(384)
75.7 (256)
(320)
76.0 (256)
76.2 (320)
(320)
77.1 (234)
uniform
(256)
75.5 (192)
75.5 (192)
75.8 (256)
75.4 (320)
75.9 (192)
DG-P3D
consecutive
(256)
77.0 (192)
77.3 (3200
77.4 (256)
(320)
77.3 (256)
78.3 (219)
netform
76.1 (192)
76.3 (128)
76.3 (256)
76.5 (256)
76.2 (192)
76.5 (192)
Table 4: The comparisons between optimization planning and hand hand-tuned strategy with DG-P3D on HMDB51
(split 1), UCF101 (splitt), Net, SS-V1, SS-V2 and Kinetics-400 datasets. The backbone is ResNet-50
pre-trained on ImageNet. The time cost for grid search/optimization planning is reported with 8 NVidia Titan V
GPUs in parallel.
Strategy
HMDB51
UCF101
ActivityNet
SS-V1
SS-V2
Kinetics-400
Hand-tuned
Top-1
53.7
86.2
74.2
51.0
62.9
77.4
Stratogy
time cost
83h
158h
166h
540b
1072h
4057h
Optimization
Iop-1
58.4
87.4
76.5
51.8
64.5
78.3
Planning
time cost
6h
13h
38h
67h
142h
288h
length, we follow the common training scheme that first learns the network with short clips and then fine-tunes the network on lengthy clips, and experiment with three strategies i - l2, l2 - la and 11+12 - lg. For each input clip length, we train the network with the same number of epochs. For learning rate decay, choose two mostly utilized strategies, i.c., 3-step learning rate decay (Wang (ct al. 2018c] Qiu et al. 2019) and cosine decay (Feichtenhofer et al. 2019p. The optimal training epoch for each strategy is determined by grid-searching from [28, 192, 256, 320, 384] epochs.
Table the comparisons between optimization planning and hand-tuned strategies with three architectures P3D, G-P3D and DG-P3D on Kinetics-400 dataset. All the networks are derived from the ResNet-50 pre-trained on ImageNet dataset. The P3D network extends the original ResNet-50 to 3D network by utilizing pseudo-3D convolutions. G-P3D and DG-P3D further employ the global context and global context plus dual-head classifier, respectively. Overall, the proposed optimization planning consistently leads to a performance boost against the best hand-tuning strategy on three networks by 1.4%, 0.9% and 0.9%, respectively. The results basically indicate the advantage of dynamically determining the training strategy. Although the number of epochs for each hand-tuned strategy is tuned by grid-search, the strategy of optimization planning is more flexible and exhibits higher performance. Moreover, with the same optimization planning strategy, DG-P3D network achieves 1.2% improvement over G-P3D, which validates the proposed dual-head classifier.
Taking our DG-P3D as 3D ConvNets, Table details the comparisons between optimization plan- ning and the hand-tuned strategy across six different datasets. The accuracy of the hand-tuned strategy is reported on the best training scheme by grid search on each dataset. Such best hand- tuned strategy can be considered as well-tuned DG-P3D model without optimization planning. The time cost of optimization planning contains the training time of exploring all the possible tran- sitions, and that of hand-tuned strategy is measured by grid-searching the candidate training strate- gies. Compared to the hand-tuned strategy, optimization planning shows consistent improvements across different datasets, and requires much less time than the exhaustive grid search due to adap- tive determination of training scheme. Figure 4 further depicts the optimal optimization paths on different datasets. An interesting observation is that SS-V1/2 tend to select uniform sampling while Kinetics-400 prefers consecutive sampling. We speculate that this may be the result of different em- phases of the two sampling strategies. In general, the most special on uniform sampling is to capture the completeness of a video with only small number of sampled frames. In contrast, consecutive sampling emphasizes the continuity in a video but may only focus on a part of the video content. The SS-VI/V2 datasets consist of fine-grained interactions and the differentiation between these interactions relies more on the completeness of an action. For example, it is almost impossible to distinguish the videos of the category "Pushing something so that it falls off the table" from those of "Pushing something so that it almost falls off but doesn 't," only based on part of the video content. In other words, uniform sampling offers the completeness of video and benefits the recognition on
7

---

Under review as a conference paper at ICLR 2021
Conncctter Samplinddo
Doroit Samphingod
Sarvehig
lead
HEAI
Best
(a) ActivityNet
(b) SS-VI
(c) SS-V2
(d) Kinetics-400
Figure 4: The best optimization path produced by the proposed optimization planning (a) ActivityNet; (h)
SS-VI; (c) SS-V2; (d) Kinetics-400.. The red edge represents the state transition in the optimization path, while the black edges denote the transitions that have been explored but not selected The optimal number of training epochs is also given for each transition in the path.
final optimization path.
SS-VI/V2. Instead, the videos in Kinetics-400 are usually with static scenes or slow motion. Hence, the completeness may not be essential in this case, but consecutive sampling encodes the continuous changes across frames and thus captures the spatio-temporal relation better.
5.4 MORE EXPERIMENTAL ANALYSIS ON OPTIMIZATION PLANNING
Next, we analyze the impact of our optimization planning from two more perspectives: 1) perfor- mance difference using different optimization paths, and the transfer of optimization path across different 31) ConvNets. With regard to the former aspect, we experiment with some variant paths on UCF101, which are built by either inserting an additional state or skipping an intermediate state in our adopted optimization path. For fair comparisons, the numbers of epochs in these variant paths are re-determined by the algorithm in Section 3.3 The results indicate that inserting and skipping one state result in an accuracy decrease of 0.2% and 0.5%% respectively. For the latter one, we conduct the experiments by utilizing the optimal path found with DG-P3D on Kinetics-400 as the path for I3D (Carreira & Zisserman 2017). Training 13D with such optimization path achieves the accuracy of 73.8% on Kinetics-400 with RGB input and leads to 1.7% performance improvement against the original 13D model. The results again demonstrate the effectiveness of our optimization planning and basically validate the generalizability of the learnt strategy across different networks.
5.5 COMPARISONS WITH STATE-OF-THE-ARY
We compare with several tfat-oof-the-rr techniques on HMDB51, UCF101 and ActivityNet datasets. The performance comparisons are summarized in Table The backbone of DG-P3D is either ResNet-50 or ResNet-101 pre-trained on ImageNet. Please note that most recent works employ Kinetics-400 pre-training improve the accuracy. Here, we also choose the two-step strat- egy that first trains DG-P3D on Kinetics-400 (K400) and then fine-tunes the network on the target dataset. The two steps are both trained with optimization planning. Overall, DG-P3D achieves the highest performances on all the three datasets, i.e., 78.8% on HMDB51, 97.8% on UCF101 and 86.8% on ActivityNet. particular, DG-P3D outperforms the other 3D ConvNets I3D, R(2+1)D, S3D-G and LGD-3D by 4.3%, 4.3%, 2.9% and 3.1% on HMDB51, respectively. The results again verify the merit of the learnt 3D ConvNets. For ActivityNet, most baselines utilize the temporal annotation to locate the foreground segment in the untrimmed videos. In our experiments, we only use the video-level annotations and our DG-P3D still surpasses the best competitor MARL by 1.1%.
Table 5: Performance comparisons with the state-of-the-art methods with RGB input (a) UCF101 (3 &IMMBI) (3 splits) and (b) AetivityNet.
(a) HMDB51 (H51) & UCFI01 (U101)
(b) ActivityNet
Method
Backbone
H51
U101
Method
Backbone
+K400 Top-1
13D C Careern Zisserman
2017
In-etcontion
74.5
95.4
TSN Wang et al 2018
BN-Inception
72.9
ARTNer Wang al 20184
BN Inception
70.9
94.3
RRA iZhu et all 2018
ResNet-152
78.8
ResNeXt (Hara (Har
2018
ResNeXi-101
70.2
94.5
MARI
Wa
2019
ResNet-112
79.8
R(2+1)D Tran et
ResNet-34
74.5
96.8
TSN
Wang ot al 20180
BN-Inssption
Y
78.9
S3D-G [Xie elal
BN-Inception
MARL Wu
SEReaNeXu152
STM [iang el al 2019
ResNei-So 50
72.2
96.2
L.GDDDD (Qui
ResNet-100
ReaNet-50
76.5
DG-P3D
ResNet-50
85.9
DG-P3D
ResNei-50
77.4
97.3
ResNet-101
77.8
ResNet- 101
78.8
97.8
ResNet-101
86.8
8

---

Under review as a conference paper at ICLR 2021
Table 6: Performance comparisons with the state-of-the-art methods with RGB input on SS-VI and SS- SS-V2.
SS-V2
Method
Backhone
Pre-train
Top-1
Top-5
Top-1
Top-5
NL MLDDD (Wang Cupla
2018)
ResNet-50
KinnetistKineticss
46.1
76.8
S3D Xie al. 2013)
BN-Inception
ImageNet
47.3
78.
TSM Lin
ResNet-50
ImageNett<ineeten
63.4
88.5
ht
Fan al
2019
ResNet-50
ImageNet
48.4
78.8
61.7
88.1
ABM-C-in
ResNet-50
ImageNet
49.8
61.2
B3D+RTTG (Nigoliciou cf
2019
ResNet-50
InageNettKinetics
49.2
78.8
GST [uup & Yunlle (019)
ResNei-50
ImageNet
48.6
77.9
62.6
87.9
STDF [Marlinez.cc 2019)
ResNet-50
ImageNet
50.1
79.5
STM (Jidng et air 2019
ResNes-50
ImageNet
50.7
80.4
64.2
89.8
DG-P3D
ResNet-50
ImageNet
51.8
81.2
64.5
90.0
Table 7: Comparisons with stale-of-the-art methods Kinetics-400 & Kinetics-600. The computational complexity is measured in GFLOPs views and the views represent the number of clips sampled from the full video during inference. * In view that it not that fair to directly compare irCSN pre-trained on IG65M (65M web videos) and other methods, here we report the performance of irCSN pre-trained on Sports IM.
Kinetles-400 (top- 1/top-/tt-5
Kinetics-600 (top- (ttp///
Method
Backbone
GFLOPs views
RGB
Hoo
Fusion
How
Fusson
B3D
BN-Inception
108 xNAA
72.1/90.3
3/86.2
75.7/92.0
R(2+1J
custom
152x115
74.3/91,4
68 68.5788.1
75.4/919
S3D-G
BN-Inception
66.4xNIA
74.7/93.4
68.0/87.6
77.2/93.
NED
ResNet-101
359 30
77.7/93.3
LGD-3D
ResNet-10r -01
195xNIA
79.4/94.4
72.3/90.9
81,2/5.2
81.5/95.6
75.092.4
83.1/96.2
X3D-XL
custom
48.4x
79.1/93.9
81.9/95.5
irCSN*
custom
96.7x30
79.0/93.5
ResNet-50
65.7x30
77.0/92.6
79.9/94.5
SlowPast
ResNet-101
213x300
78.9/93.5
81. 3/951
ResNet-101-NL
234x30
79.8/93.9
81 8/981
DG-P3D
ResNet-50
x0
78.3/93.7
72.0/90.9
80. 80.3/4477
4/95.6
8/93.2
82.7/95.9
ResNet-101
218x30
80,4/94.7
73.2/91.2
82.5/96.0
82.6/9599
76.7/93.4
84.3/96.6
Then, we turn evaluate DG-P3D with optimization planning on four large-scale datasets, i.e., SS- V1, SS-V2, Kinetics-400 and Kinetics-600, The top-1 and top-5 accuracies are reported on the four datasets. For Kinetics datasets, we additionally consider the flow modality for fair comparison with the baselines. The two-direction optical flow image is extracted by TV-L1 algorithm (Zach et al,, 2007] in this paper. To reduce the time cost, the best optimization path found on the RGB modality of Kinetics-400 is utilized as the path for flow modality modalities on Kinetics-600. The results are shown in Table 6 and Table 7 Specifically, DG-P3D achieves the best performance with top-1 accuracy of 51.8% on SS-V1 and 64.5% VS-V2. DG- P3D is superior to STM, which reports the best known results, by 1.1% and 0.3% respectively. On Kinetics-400, with only RGB input, DG-P3D achieves 80.4% top-1 accuracy, which makes the im- provements over the recent 3D ConvNets irCSN (Tran et al. 2019). X3D-XL (Feichtenhofer 2020 LGD-3D QQu et al. 2019). SlowFast (Feichtenhofer et al. 2019 by 1.4%, 1.3%, 1.0% and 0.6%. respectively. Such accuracy is even higher than that of the two-stream I31) (Carreira & Zisserman 2017]. R(2+1)D (Tran et al.] 2018) and S3D-G (Xie et al., 2018). When fusing the prediction from both modalities, the accuracy of DG-P3D is further improved to 82.5%. The similar performance trends are also observed on Kinetics-600. The two-stream DG-P3D achieves 84.3% top-1 accuracy, which leads the performance by 1.2% against the best competitor of two-stream LGD-3D.
Kinetics-400, and for both RGB and flow
CONCLUSION
We have presented optimization planning which aims to automate the training scheme of 3D Con- vNets. Particularly, a training process is decided by sequence of training states, namely optimiza- tion path, plus the number of training epochs for each state. We specify the hyper-parameters in each state and the permutation of states determines the changes of hyper-parameters. Technically, we propose a dynamic programming method to seek the best optimization path in the pre-defined candidate transition graph and each state transition is stopped adaptively by estimating the knee point on the performance-epoce curve. Furthermore, we devise a new 3D ConvNets, i.e., DG-P3D, with unique design of the dual-head classifier. The results on seven video benchmarks, which are different in terms of data scale, target categories and video duration, validate our proposal. Notably, DG-P3D with optimization planning obtains superior performances all the seven datasets.

---

Under review as a conference paper at ICLR 2021
REFERENCES
Mary Ann Branch, Thomas F. Coleman, and Yuying Li. A subspace, interior, and conjugate gradient method for large-scale bound-constrained minimization problems. SIAM Journal on Scientific Computing, 21:1-23, 1999.
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
Yue Cao. Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet squeeze-cxcitation networks and beyond. In ICCV Workshop, 2019.
João Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
João Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. ArXiv,
014,0, 2018.
Jerry Chee and Panos Toulis. Convergence diagnosties for stochastic gradient descent with constant learning rate. In AISTATS, 2018.
Quanfu Fan, Chun-Fu Chen, Hilde Kuchne, Marco Pistoia, and David Cox. More is less: Learn- ing efficient video representations by big-little network and depthwise temporal aggregation. In NeurlPS, 2019.
Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In CVPR, 2020.
Christoph Feichtenhofer, Axel Pinz, and Andrew Zisserman. Convolutional two-stream network fusion for video action recognition. In CVPR, 2016.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In ICCV, 2019.
Rong Ge, Sham M. Kakade, Rahul Kidambi, and Praneeth Netrapalli. The step decay schedule: near optimal, geometrically decaying learning rate procedure. ArXiv, abs/1904. 12838, 2019.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska. Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fründ, Peter Yianilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet. In CVPR, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Shuiwang Ji, Wei Xu, Ming Yang. and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE Trans. on PAMI, 35(1):221-231, 2013.
Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Ser- gio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embed- ding. In ACM MM, 2014.
Boyuan Jiang, Mengmeng Wang, Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In ICCV, 2019.
Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei- Fei. Large-scale video classification with convolutional neural networks. In CVPR, 2014.
H. Kuchne, H. Jhuang, Garrote, T. Poggio, and T. Serre. HMDB: a large video database for human motion recognition. In ICCV, 2011.
10

---

Under review as a conference paper at ICLR 2021
Hunter Lang, Lin Xiao, and Pengchuan Zhang. Using statistics automate stochastic optimization. In NeurlPS, 2019.
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In ICCV, 2019.
Chenxu Luo and Alan L.. Yuille. Grouped spatial-temporal aggregation for efficient action recogni- tion. In ICCV, 2019.
Brais Martínez, Davide Modolo, Yuanjun Xiong, and Joseph Tighe. Action recognition with spatial- temporal discriminative filter banks. In ICCV, 2019.
Andrei Liviu Nicolicioiu, Iulia Duta, and Marius Leordeanu. Recurrent space-time graph neural networks. In NeurlPS, 2019.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In ICCV, 2017.
Zhaofan Qiu, Ting Yao, Chong-Wai Wang Ngo, Xinmei Tian, and Tao Mei. Learning spatio-temporal representation with local and global diffusion. CVPR, 2019.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. In NeurIPS, 2014.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. UCF101: A dataset of 101 human action classes from videos in the wild. CRCV-TR-12-01, 2012.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In ICCV, 2015.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look at spatiotemporal convolutions for action recognition. In CVPR, 2018.
Du Tran, Heng Wang. Lorenzo Torresani, and Matt Feiszli. Video classification with channel- separated convolutional networks. In JCCV, 2019.
Gül Varol, Ivan Laptev, and Cordelia Schmid. Long-term temporal convolutions for action recogni- tion, IEEE Trans. on PAMI, 1510)):510-1111111 2018.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Lue Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
Limin Wang, Weiqi Li. Wen Li, and Luc Van Gool. Appearance-and-rrlaann networks for video classification. In CVPR, 2018a.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition in videos. IEEE Trans. on PAMI, 2018b.
Xiaolong Wang and Abhinav Gupta. Videos as space-time region graphs. In ECCV, 2018.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018c.
Chao-Yual Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. A multigrid method for efficiently training video models. In CVPR, 2020.
Wenhao Wu, Dongliang He, Xiao Tan, Shifeng Chen, and Shilei Wen. Multi-agent reinforcement learning based frame sampling for effective untrimmed video recognition. In ICCV, 2019.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotem- poral feature learning: Speed-accuracy trade-offs in video classification. In ECCV, 2018
Sho Yaida. Fluctuation-diisipattoo relations for stochastic gradient descent. In ICLR, 2019.
11

---

Under review as a conference paper at ICLR 2021
C Zach, T Pock, and H Bischof. A duality based approach for realtime tv-l1 optical flow. Pattern Recognition, 2007.
Chen Zhu, Xiao Tan, Feng Zhou, Xiao Liu, Kaiyu Yue, Errui Ding, and Yi Ma. Fine-grained video categorization with redundancy reduction attention. In ECCV, 2018.
Xinqi Zhu, Chang Xu, Langwen Hui, Cewu Lu, and Dacheng Tao. Approximated bilinear modules for temporal modeling. In ICCV, 2019.
A
APPENDIX
The appendix contains: 1) the collection of the performance-epoch curves; 2) the comparisons of 3D ConvNets architectures; 3) visualization of more optimization paths by optimization planning.
A.1 CURVE COLLECTION
To evaluate the functions for knee point estimation, we pre-collect 162 performance-epoch curves from the training processes of different networks on different datasets. Particularly, P3D, G-P3D and DG-P3D networks are trained using hand-tuned strategy on six different datasets, i.e., HMDB51, UCF101, ActivityNet, SS-VI SS-V2 and Kinetics-400. To obtain the curves with different settings, for each dataset, the network is firstly trained with 8-frame clips using 3-step learning rate strategy ([0.01, 0.001, J). and then fine-tuned on 16-frame and 32-frame clips. Each step is trained for 50 epochs. Therefore, we collect 3 x 6 9 162 curves in total.
A.2 3D CONvNETS ARCHITECTURES
Table 8: Computational cost of different 3D ConvNets architecture.
Network
BBD
R(2+1)D
LGD-3D
SlowFast
P3D
G-P3D
DG-P3D
GFLOPs
108
152
195
234
196
203
218
cobpeees the computational cost of different network architectures. The number floating- point operations (FLOPs) for crop is given each network. Overall, P3D network used in this paper requires similar computations with LGD-3D. The global context and dual-head classifier leads to 3.5% and 7.3% additional computations, respectively. Ultimately, the one-crop prediction of DG-P3D spends 218 GFLOPs, which still lower than that of SlowFast network. The results basically indicate that our DG-P3D potentially more economic and effective.
A.3 VISUALIZATION MORE STRATEGIES
Figure 5 depicts the best optimization paths learnt by optimization planning HMDB51, UCF101, ActivityNet, SS- V1, 11, SS- V2 and Kinetics-400, respectively. We additionally show the best optimiza- tion paths HMDB51 and UCF101 in Figure 5(g) and Figure ((h) when taking Kinetics-400 for network pre-training. As expected, the training strategy predicted by optimization planning changes in response to different network initialization.
12

---

Under review as a conference paper at ICLR 2021
Sampling
Unlforn Samppiing (ut)
Sampling (20
INSHI
FIW
Sumppiing
Est
Stet
Ved
HMDB51
(b) UCF101
(c) ActivityNel
(d) SS-VI
Connecative Sampling led
Sampling
REW
Sampping IAO
S
Sumpping
Indal
Pune
Der:
2011
(e) SS-V2
(I) Kinetics-400
(g) HMDB51 (+K400) (h) UCFI01 (+K400)
Figure 5: The best optimization path produced by the proposed optimization planning (a) HMDB51, (b) UCF101, (c) Neti (d) SS-V1, (e) SS-V2, (f) Kinetics-400, (g) HMDB51 with Kinetics-400 pre-training and (h) UCF101 with Kinetics-400 pre-training. The red edge represents the state transition in the optimization path and the optimal number of training epochs is also given for each transition.
13


Reviews:
[{"review_id": "zPVNza_QRK", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The overall quality is good. It addresses the training issue of 3D convolutional network in a novel perspective and shows promising results. The presentation is clear except for some grammar mistakes. \n\nDetailed Comments:\n\n- In Sec. 3.2, how is the transition performance evaluated is not clear. What is the cost / gain from i to j? Is it the same for all states?\n\n- Ablation experiments should be provided to show the performance difference using different paths. Is the adopted path actually the optimal path? And how about if some intermediate states are skipped?\n\n- The proposed optimization planning doesn't seem to work specifically for the 3D convnets and for the action recognition task. As long as we need to train a network in stages with multiple hyper-parameters, this should apply theoretically. \n\n- Abstract: 'follows' --> 'is followed'", "extra_scores": {}, "extra_text": {}}, {"review_id": "hZuFQf1rEaL", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Overview:\nThe paper proposes a novel way to automatically tune 3D ConvNet hyper-parameters (learning rate, input clip length, sampling way). This is achieved by decomposing the optimization path into several states and the state transition is triggered when the knee-point on the performance-epoch curve is met. Extensive experiments are conducted on popular video benchmarks and show that the optimization planning is effective to improve the accuracy and requires less time time compared to the hand-tuned procedure.\n\n\nStrengths:\n\n++ The paper provides a novel perspective of designing the hyper-parameters for 3D ConvNets. The automatic planning through the optimization path alleviate the tedious work of hand tuning and save a lot of times: Once the knee-point on the performance-epoch curve is achieved, we can easily transit to the next optimization state (with another set of hyper-parameters).\n\n\n++ The experiments are thorough. The authors conduct experiments on all popular video recognition benchmarks. Also the experiment results corroborate the effectiveness of optimization planning. \n\nWeaknesses:\n\n-- The construction of transition graph has some strict principles that could be relaxed. \n  (1) For scheduling learning rate, It only considers that \"training starts from high learning rate\". However, previous works (e.g. ICLR2017: https://arxiv.org/abs/1608.03983) have shown that cyclic cosine learning schedule might benefit training. Therefore, I don't see the necessity that we put this restriction.\n  (2) For scheduling input length, this is not the first time that people do this. For example, Wang et. al. (CVPR2018, https://arxiv.org/abs/1711.07971) first train using 64 frames as input and then finetune on 128 frames. Also, Wu et. al. (CVPR2020, https://arxiv.org/abs/1912.00998) proposes another way of adjust the sampling strategy.\n\n\n-- The comparison is mostly conducted on the newly proposed DG-P3D family architectures. It would be better if there is an additional row of results on the standard I3D/SlowFast network in Table 3. (this is somewhat minor and not required considering the time limit)\n\n\n-- \"82.5%\" (with flow stream) is not the \"new record\" for Kinetics-400. ir-CSN-152 (ICCV2019, https://arxiv.org/abs/1904.02811) can achieve 82.6% on Kinetics-400 using **RGB only**. OmniSource (ECCV2020, https://arxiv.org/abs/2003.13042) improves it to 83.6% using additional data.\n\n\n\n\nI have some additional questions:\n\n-- In Figure 4, it shows that SS-V1/2 would prefer uniform sampling however Kinetics would prefer consecutive sampling. However, this seems somewhat counter-intuitive to me. We know that Kinetics have videos with static scenes or slow motion while SS-V1/2 have some videos with more intensive motion. Does it make more sense to use dense frames (consecutive) to recognize intensive motion?", "extra_scores": {}, "extra_text": {}}, {"review_id": "_BiKiPSl17q", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "- The paper proposed a Dual-head Global-contextual Pseudo-3D (DG-P3D) network and an automated optimization path to train 3D ConvNet for action recognition. The proposed method is evaluated on various of action recognition datasets, and achieved convincing results.\n\n- Another contribution of the paper is its decreased cost. It mentioned that the time cost for grid optimization planning with 8 NVidia Titan V GPUs reduced the running time from 4057h to 288h on Kinetics.\n\n- The optimization method is proposed for 3D ConvNet, However, the authors only benchmarked it with the new proposed DG-P3D network. It could be interesting to know what is the performance applying it to other basic 3D ConvNets. It is also interesting to know what is the performance of DG-P3D without this new optimization strategy.\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "YsZuEHtGnK7", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n  This paper proposed two things:  \n    1. hyper-parameter planning for training action recognition models.   \n    2. a new 3D net architecture for action recognition. \n\n  The results show with the planning, authors can reduce training time significantly. and the proposed DG-P3D also is good for action recognition\n\n\nStrong points:\n\n  1. This paper explores a new approach for hyperparameter tuning to get a better model quickly.\n\n  2. This paper performs extensive experiments to validate their proposed methods.\n\nWeak points:\n\n  1. It misses one very important reference on the same topic, \"Multigrid Method for Efficiently Training Video Models\" CVPR 2020. This paper also proposes a new way to efficiently train action recognition models. \n\n  2. The way authors compared to others are mixed, making that it is a little bit difficult to compare the contribution of the paper. \n  \n    2.1 To be more fair compare to SOTA approaches in Table 4 and Table 5 with respect to the contributions of DG-P3D, the author should include the results without planning approach and use a similar training setting to the other SOTA approaches. Or apply the proposed optimization planning on those SOTA approaches to see the improvement. In this case, the advantages of each component will be more clear. \n\n    2.2 When comparing to other approaches, authors should either include the number of frames used or computational load (like FLOPs) into the consideration, instead of performance only. E.g., for Kinetics400, slowfast-ResNet-50 (77.0% top-1 acc) only use 32 frames while the proposed models for Kinetics400 use 128 frames (If I understand correctly); thus, simply comparing the accuracy is unfair. On the other hand, If authors want to compare the performance only, authors might want to include the model with different backbones, e.g. bLVNet-TAM has models achieved 53.1% on SS-V1 and 65.2% on SS-V2 with RGB only.\n\n\nQuestions:\n\n  1. If a node has multiple destinations, e.g. node S5 in Figure 1a, how do the author select S7 or S8? In my understanding, the proposed algorithm will train the model at S5 with the parameters at S7 and S8 independently and then select the better one. Is it correct?\n\n  2. Followed by 1, when encountering multiple destinations, if all destination nodes need to be explored before making the transition, do the author takes those time into account as well in Table 4? I do not find those in Figure 4, either.\n\n  3. What is the delay parameter T in practical? Is it a dataset-dependent parameter?\n\n  4. What is the inference protocol used in the evaluation? The authors mentioned there are two common approaches but do not mention which one is used.\n\nOthers:\n\n 1. When comparing to SOTA, I think authors miss \"X3D: Expanding Architectures for Efficient Video Recognition\" CVPR 2020.\n\nAfter rebuttal:\n\nThe authors addressed some of my concerns, and the proposed optimization plan is interested.\n\n1. I still think that the proposed architecture does not outperform SOTA architecture, like SlowFast. \n  - First, in the rebuttal, the authors mention that with hand-crafted strategies, the proposed model is 0.4% better than SlowFast; nonetheless, the hand-crafted strategies train longer epochs than SlowFast. (SlowFast is trained with 196 epochs.) As authors propose new strategies, they know better how different strategies could affect. \n  - Second, in the rebuttal, the authors improve I3D by 1.7% with optimization planning, which means, SlowFast might outperform the proposed model if involving optimization planning. Moreover, in the revised Table 7, the SlowFast does not have more FLOPs than the proposed network under the same backbone. Simply checking Table 8 might be confused. \n\n2. About the optimization planning, it is still confused about how many epochs spent on the explored nodes. E.g. in Figure 4, for the Kinetics dataset, I do not understand why there is a black edge between S4 and S5 as S4 is not reached. (Authors noted that the black edges mean the explored strategies.) And why not there is no number on those explored edges? Another example, if we look at Table 3 and Figure 4 together, and again for the Kinetics dataset, the summation of those red numbers is 248, but this number is larger than any number in Table 3. As the authors mention that they included the epochs of the explored strategies in Table 3, that means they only spend 29 epochs in the exploration (if we treat Figure 4 for DG-P3D in Table 3.). For me, it seems too good to be true.\n\nThus, I keep my rating. \n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the idea of hyperparameter planning and the thorough experimentation. Some concerns remain regarding the comparison between this method and SlowFast that require to be addressed. Also, the scope of the paper that targets hyperparameter optimization networks for action recognition specifically, may be too narrow for an ICLR audience. 
Decision: Reject