Optimization Planning for 3D ConvNets

1 INTRODUCTION . The recent advances in 3D Convolutional Neural Networks ( 3D ConvNets ) have successfully pushed the limits and improved the state-of-the-art of video recognition . For instance , an ensemble of LGD3D networks ( Qiu et al. , 2019 ) achieves 17.88 % in terms of average error in trimmed video classification task of ActivityNet Challenge 2019 , which is dramatically lower than the error ( 29.3 % ) attained by the former I3D networks ( Carreira & Zisserman , 2017 ) . The result basically indicates the advantage and great potential of 3D ConvNets for improving the performance of video recognition . Despite these impressive progresses , learning effective 3D ConvNets for video recognition remains challenging , due to large variations and complexities of video content . Existing works on 3D ConvNets ( Tran et al. , 2015 ; Carreira & Zisserman , 2017 ; Tran et al. , 2018 ; Wang et al. , 2018c ; Feichtenhofer et al. , 2019 ; Qiu et al. , 2017 ; 2019 ) predominately focus on the designs of network architectures but seldom explore how to train a 3D ConvNets in a principled way . The difficulty in training 3D ConvNets originates from the high flexibility of the training scheme . Compared to the training of 2D ConvNets ( Ge et al. , 2019 ; Lang et al. , 2019 ; Yaida , 2019 ) , the involvement of temporal dimension in 3D ConvNets brings two new problems of how many frames should be sampled from the video and how to sample these frames . First , the length of video clip is a tradeoff to control the balance between training efficiency and long-range temporal modeling for learning 3D ConvNets . On one hand , training with short clips ( 16 frames ) ( Tran et al. , 2015 ; Qiu et al. , 2017 ) generally leads to fast convergence with large mini-batch , and also alleviates the overfitting problem through data augmentation brought by sampling short clips . On the other hand , recent works ( Varol et al. , 2018 ; Wang et al. , 2018c ; Qiu et al. , 2019 ) have proven better ability in capturing long-range dependency when training with long clips ( over 100 frames ) at the expense of training time . The second issue is the sampling strategy . Uniform sampling ( Fan et al. , 2019 ; Jiang et al. , 2019 ; Martı́nez et al. , 2019 ) offers the network a fast-forward overview of the entire video , while consecutive sampling ( Tran et al. , 2015 ; Qiu et al. , 2017 ; 2019 ; Varol et al. , 2018 ; Wang et al. , 2018c ) can better capture the spatio-temporal relation across frames . Given these complex choices of training scheme , learning a powerful 3D ConvNets often requires significant engineering efforts of human experts to determine the optimal strategy on each dataset . That motivates us to automate the design of training strategy for 3D ConvNets . In the paper , we propose optimization planning mechanism which seeks the optimal training strategy of 3D ConvNets adaptively . To this end , our optimization planning studies three problems : 1 ) choose between consecutive or uniform sampling ; 2 ) when to increase the length of input clip ; 3 ) when to decrease the learning rate . Specifically , we decompose the training process into several training states . Each state is assigned with the fixed hyper-parameters , including sampling strategy , length of input clip and learning rate . The transition between states represents the change of hyper-parameters during training . Therefore , the training process can be decided by the permutation of different states and the number of epochs for each state . Here , we build a candidate transition graph to define the valid transitions between states . The search of the best optimization strategy is then equivalent to seeking the optimal path from the initial state to the final state on the graph , which can be solved by dynamic programming algorithm . In order to determine the best epoch for each state in such process , we propose a knee point estimation method via fitting the performance-epoch curve . In general , our optimization planning is viewed as a training scheme controller and is readily applicable to train other neural networks in stages with multiple hyper-parameters . To the best of our knowledge , our work is the first to address the issue of optimization planning for 3D ConvNets training . The issue also leads to the elegant view of how the order and epochs for different hyper-parameters should be planned adaptively . We uniquely formulate the problem as seeking an optimal training path and devise a new 3D ConvNets with dual-head classifier . Extensive experiments on seven datasets demonstrate the effectiveness of our proposal , and with optimization planning , our 3D ConvNets achieves superior results than several state-of-the-art techniques . 2 RELATED WORK . The early works using Convolutional Neural Networks for video recognition are mostly extended from 2D ConvNets for image classification ( Karpathy et al. , 2014 ; Simonyan & Zisserman , 2014 ; Feichtenhofer et al. , 2016 ; Wang et al. , 2016 ) . These approaches often treat a video as a sequence of frames or optical flow images , and the pixel-level temporal evolution across consecutive frames are seldom explored . To alleviate this issue , 3D ConvNets in Ji et al . ( 2013 ) is devised to directly learn spatio-temporal representation from a short video clip via 3D convolution . Tran et al . design a widely-adopted 3D ConvNets in Tran et al . ( 2015 ) , namely C3D , consisting of 3D convolutions and 3D poolings optimized on the large-scale Sports1M ( Karpathy et al. , 2014 ) dataset . Despite having encouraging performances , the training of 3D ConvNets is computationally expensive and the model size suffers from a massive growth . Later in Qiu et al . ( 2017 ) ; Tran et al . ( 2018 ) ; Xie et al . ( 2018 ) , the decomposed 3D convolution is proposed to simulate one 3D convolution with one 2D spatial convolution plus one 1D temporal convolution . Recently , more advanced techniques are presented for 3D ConvNets , including inflating 2D convolutions ( Carreira & Zisserman , 2017 ) , non-local pooling ( Wang et al. , 2018c ) and local-and-global diffusion ( Qiu et al. , 2019 ) . Our work expands the research horizons of 3D ConvNets and focuses on improving 3D ConvNets training by adaptively planning the optimization process . The related works for 2D ConvNets training ( Chee & Toulis , 2018 ; Lang et al. , 2019 ; Yaida , 2019 ) automate the training strategy via only changing the learning rate adaptively . Our problem is much more challenging especially when temporal dimension is additionally considered and involved in the training scheme of 3D ConvNets . For enhancing 3D ConvNets training , the recent works ( Wang et al. , 2018c ; Qiu et al. , 2019 ) first train 3D ConvNets with short input clips and then fine-tune the network with lengthy clips , which balances training efficiency and long-range temporal modeling . The multigrid method ( Wu et al. , 2020 ) further cyclically changes spatial resolution and temporal duration of input clips for a more efficient optimization of 3D ConvNets . The research in this paper contributes by studying not only training 3D ConvNets with multiple lengths of input clips , but also adaptively scheduling the change of input clip length through optimization planning . 3 OPTIMIZATION PLANNING . 3.1 PROBLEM FORMULATION . The goal of optimization planning is to automate the learning strategy of 3D ConvNets . Formally , the optimization process of 3D ConvNets can be represented as an optimization path P = 〈S0 , S1 , ... , SN 〉 , which consists of one initial state S0 and N intermediate states . Each intermediate state is assigned with the fixed hyper-parameters , and the training is performed with these N different settings one by one . The training epoch on each setting is decided by T = { t1 , t2 , ... , tN } , in which ti denotes the number of epochs when moving from Si−1 to Si . The hyperparameters include sampling strategy ∈ { cs , us } , length of input clip ∈ { l1 , l2 , ... , lNl } and learning rate ∈ { r1 , r2 , ... , rNr } , where cs and us denotes consecutive sampling and uniform sampling , respectively . In this case , there are 2×Nl ×Nr valid types of training states . The objective function of optimization planning is to seek the optimal strategy { P , T } by maximizing the performance of the final state SN : maximize P , T V ( SN ) , ( 1 ) where V ( · ) is the target performance , i.e. , mean accuracy on validation set in our case . 3.2 OPTIMIZATION PATH . To plan the optimal permutation of training states , we first choose a final state SN , which is usually with low learning rate and lengthy input clip . Then , the problem of seeking an optimal optimization path to SN is naturally decomposed to the subproblem of finding the optimization path to an intermediate state Si and the state transition from Si to SN . As such , the problem can be solved by dynamic programming . Specifically , the solution of optimization path P ( SN ) can be given in a recursive form : P ( SN ) = 〈P ( Si∗ ) , SN 〉 , i∗ = argmax i { V ( Si → SN ) } . ( 2 ) When executing the transfer from the state Si to the state SN , we fine-tune the 3D ConvNets at the state Si by using the hyper-parameters at the state SN . We then evaluate such fine-tuned model on the validation set to measure the priority of this transition , i.e. , V ( Si → SN ) . We choose the state S∗i , which achieves the highest priority of transition to the state SN , as the preceding state of SN . In other words , the optimal path for SN derives from the best-performing preceding state Si∗ . Here , we propose to pre-define all the valid transitions in a directed acyclic graph and determine the best optimization path of each state one by one in the topological order . Figure 1 ( a ) shows one example of the pre-defined transition graph . In the example , we set the number of candidate input clip lengths Nl = 3 and the number of candidate learning rates Nr = 3 . Hence , there are 2× 3× 3 = 18 candidate states . Then , the possible transitions , i.e. , the connections between states , are determined by the following principles : ( 1 ) The transitions between states with different sampling strategies are forbidden . We choose S9 and S18 as the final states for consecutive sampling and uniform sampling , respectively . ( 2 ) The training only starts from high learning rate and short input clips . ( 3 ) The intermediate state can be only transferred to a new state , where either the learning rate is decreased or the length of input clip is increased in the new state . Please note that , some very specific learning rate strategies , e.g. , schedules with restart or warmup , show that increasing the learning rate properly may benefit training . Nevertheless , there is still no clear principle of when to increase the learning rate , and thus it is very difficult to automate these schedules . In the works of adaptively changing the learning rate for 2D ConvNets training ( Ge et al. , 2019 ; Lang et al. , 2019 ; Yaida , 2019 ) , such cyclic schedules are also not taken into account . As a result , we only consider the schedule of decreasing learning rate in the transition graph . These principles can simplify the transition graph and reduce the time cost when solving Equ. ( 2 ) . We take this graph as basic transition graph . Furthermore , we also build an extended transition graph by enabling simultaneously decreasing the input clip length and the learning rate , as shown in Figure 1 ( b ) . In such graph , the training strategies are more flexible .

Summary: The paper proposes a novel way to automatically tune 3D ConvNet hyper-parameters (learning rate, input clip length, sampling way). This is achieved by decomposing the optimization path into several states and the state transition is triggered when the knee-point on the performance-epoch curve is met. Extensive experiments are conducted on popular video benchmarks and show that the optimization planning is effective to improve the accuracy and requires less time time compared to the hand-tuned procedure.