Low-Fidelity Video Encoder Optimization for Temporal Action Localization

1 Introduction . Video analysis has become an increasingly important area of research , encompassing multiple relevant problems such as action recognition [ 8 , 13 ] , temporal action localization [ 68 , 7 , 12 , 22 , 59 , 60 , 55 , 67 , 15 ] , video grounding [ 36 , 65 , 64 , 66 , 49 , 16 , 24 ] , and video question answering [ 25 , 31 ] . Among those , temporal action localization ( TAL ) [ 68 , 22 ] is a fundamental task , as natural videos are not temporally trimmed . Given an untrimmed video , TAL aims to identify the start and end points of all action instances and recognize their category labels simultaneously . A typical TAL model is based on deep convolutional neural networks ( CNNs ) composed of two modules : a video encoder and a TAL head . The video encoder is often shared across different TAL methods ( e.g. , G-TAD [ 60 ] , BC-GNN [ 3 ] ) ∗Work done during an internship at Samsung AI Centre . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . by taking a specific off-the-shelf action classification model ( e.g. , C3D [ 51 ] , I3D [ 8 ] , TSM [ 33 ] ) , with the differences residing only in the TAL head . However , instead of short ( e.g. , 10 seconds ) trimmed video clips as in action recognition , the input videos to a TAL model are characterized by much longer temporal duration ( e.g. , 120 seconds ) . This causes unique computational challenges that remain unsolved , particularly in model optimization . In standard optimization of a TAL model , a two-stage transfer learning pipeline is often involved : 1 . First , the video encoder is optimized on a large source video classification dataset ( e.g. , Kinetics [ 28 ] ) and , optionally , finetunned on the trimmed version of the target dataset under action classification supervision ; 2 . Second , the video encoder is frozen and the TAL head is optimized on the target action localization dataset ( e.g. , ActivityNet [ 22 ] , HACS [ 68 ] ) under TAL task supervision . With this widely-used TAL training pipeline , the video encoder is only optimal for action classification but not for the target TAL task . Specifically , the video encoder is trained so that different short segments within an action sequence are mapped to similar outputs , thus encouraging insensitivity to the temporal boundaries of actions . This is not desirable for a TAL model . We identify this as a task discrepancy problem . Consequently , the final TAL model could suffer from suboptimal performance . Indeed , jointly optimizing all components of a CNN architecture end-to-end with the target task ’ s supervision is a common practice , e.g. , training models for object detection in static images [ 18 , 46 , 38 ] . Unfortunately , this turns out to be non-trivial for TAL . As mentioned above , model training is severely restricted by the large input size of untrimmed videos and subject to the memory constraint of GPUs . This is why the two-stage optimization pipeline as described above becomes the most common and feasible choice in practice for optimizing a TAL model . On the other hand , existing transfer learning methods mostly focus on tackling the data distribution shift problem across different datasets [ 70 , 50 ] , rather than the task shift problem we study here . Regardless , we believe that solving this limitation of the TAL training design bears a great potential for improving model performance . In this work , we present a simple yet effective low-fidelity ( LoFi ) video encoder optimization method particularly designed for better TAL model training . It is designed to adapt the video encoder from action classification to TAL whilst subject to the same hardware budget . This is achieved by introducing a simple strategy characterized by a new intermediate training stage where both the video encoder and the TAL head are optimized end-to-end using a lower temporal and/or spatial resolution ( i.e. , low-fidelity ) in the mini-batch construction . Compared to the standard training method , our proposed strategy does not increase the GPU memory standard ( often a hard constraint for many practitioners ) . Crucially , with our LoFi training the gradients back-propagate to the video encoder from a temporal action localization loss whilst conditioned on the target TAL head , enabling the learning of a video encoder sensitive to the temporal localization objective . We make the following contributions in this work . ( 1 ) We investigate the limitations of the standard optimization method for TAL models , and consider that the task discrepancy problem hinders the performance of existing TAL models . Despite it being a significant ingredient , video encoder optimization is largely ignored by existing TAL methods , left without systematic investigation . ( 2 ) To improve the training of TAL models , we present a novel , simple , and effective low-fidelity ( LoFi ) video encoder optimization method . It is designed specifically to address the task discrepancy problem with the TAL model ’ s video encoder . ( 3 ) Extensive experiments show that the proposed LoFi optimization method yields new state-of-the-art performance when combined with off-the-shelf TAL models ( e.g. , G-TAD [ 60 ] ) . Critically , our method achieves superior efficiency/accuracy trade-off with clear inference cost advantage and good generalizability to varying-capacity video encoders . 2 Related Work . Temporal action localization ( TAL ) models : TAL models can be grouped by architectural design pattern into two categories , one-stage and two-stage architectures . One-stage methods , either predict temporal action boundaries or generate proposals , and classify them within the same network [ 3 , 9 , 21 , 40 , 34 , 58 , 60 , 62 ] . The latter type , two-stage models , generate sets of action proposals ( e.g. , segments ) [ 6 , 12 , 14 , 23 , 39 ] and then an auxiliary head is used for classification of each proposal into an action class [ 35 , 47 , 48 , 63 , 69 ] . In this work , rather than introducing a novel model design , we focus on the training of generic TAL models , with a particular aim to improve the video encoder optimization . This is a relatively less investigated aspect in the TAL literature . Video encoders in TAL : The video encoder is an indispensable part of a TAL model . Main design choices include the base architecture of video encoder and its optimization procedure . With regards to the architecture , the two-stream Temporal Segment Network ( TSN ) [ 54 ] is one of the most common video encoders in existing TAL methods [ 3 , 34 , 35 , 60 ] . Concretely , these works use two TSN networks , one with a ResNet50 [ 20 ] backbone trained on RGB video frames , and the other with a BN-Inception backbone [ 26 ] trained on optical flow . Other alternatives used as a video encoder for TAL include two-stream I3D model [ 8 ] ( see [ 19 , 63 ] ) and Pseudo-3D [ 45 ] ( see [ 40 ] ) . In terms of optimization , a typical paradigm is two-staged : first pre-training the video encoder and then , in a second stage , training the TAL head of the model with the video encoder fixed . This is constrained by the inherent hardware budget derived from having a large per-video input size . In particular , the video encoder is pre-trained using a cross-entropy loss for action recognition on a large-scale video classification dataset such as Kinetics [ 28 , 64 ] . An optional step is to further pre-train it on the foreground segments of the target TAL dataset [ 34 , 60 , 11 , 44 ] . This brings a mismatch between training and inference for the video encoder , which we call a task discrepancy problem . More specifically , although trained to distinguish the content of different action classes , the video encoder is less sensitive to action temporal boundaries and thus less effective for the TAL task . In fact , due to their inherent design , CNNs have limited localization capabilities [ 41 ] , unless they are augmented with specialized localization-specific layers [ 37 ] . Additionally , the action classification task focuses only on the foreground content whilst ignoring per-class background segments , including the transition between foreground and background . In this paper we propose a novel low-fidelity video encoder pre-training method to solve this limitation with existing TAL methods . While current TAL literature mostly relies on pre-training through supervised learning , the rapid advancement of self-supervised learning makes it a promising alternative [ 2 , 4 , 42 , 43 , 56 ] . Some works have focused on finding effective temporal-related pretext tasks , from frame ordering learnt through triplets of frames [ 43 ] , to sorting the frames of a sequence [ 30 ] , distinguishing whether sequences are played forward or backwards [ 56 ] or through playback speed-related pretext tasks [ 4 , 61 , 53 , 27 ] . These methods exploit video-specific characteristics to force the network to focus on some sort of semantic content within video , inducing representations capturing long-term temporal semantic relations , but force invariance to or ignore the relative positioning of the snippets within the action instances . They are thus not suited for pre-training the video encoder of a TAL model . Very recent works [ 1 , 59 ] have exploited some of the aforementioned techniques for better pre-training of action localization models . For example , localization-tailored data augmentation and classification is adopted by [ 59 ] . However , these works introduce a large amount of extra video data and additional stream networks , both of which are expensive in terms of memory and computation . In contrast , our method aims to improve TAL modelling directly without the need for learning from extra training video data and using an expensive second network , nor relying on optical flow obtained at high computational cost . 3 Method . A TAL model takes as input a long untrimmed video with a varying number of frames . For design convenience , it is typical to represent a varying-length video by decomposing it into a fixed-length sequence of L snippets . The definition of a snippet is the same as in action recognition , where first a number of consecutive frames ( e.g. , 64 ) is selected and then sub-sampled with stride r ( e.g. , stride 8 to obtain 8-frame snippets ) . To represent a snippet , one first applies a video encoder to extract frame-level feature vectors and then averages them to obtain the snippet-level feature representation [ 6 , 12 , 17 , 35 ] . The resulting snippet feature sequence is denoted as X ∈ RC×L , where C is the feature dimension of each snippet , and L is the number of snippets . In the training set , each video is associated with its ground truth , consisting of a set of action instance annotations Ψ . In particular , each action instance is represented as a segment , each including the start time , the end time , and the action class label . The objective is to train a TAL model that can accurately localize all the target action instances in a given untrimmed video . To that end , the model predicts a varying number of action instances Φ , each comprised of the predicted temporal boundaries , action class , and confidence score .

Summary: ** Update 8/16/21 **   Thanks to the authors for their rebuttal in response to my questions/concerns. After reading the rebuttal and the other reviews, I am updating my score to 6, to better reflect the contributions made by the work.  **   The paper aims to tackle the hardware constraint for end-to-end optimization of both video encoding and TAL head models. Different from prior works that freeze the video encoding network when training the TAL head, the proposed method optimizes both of them jointly on low-resolution (spatially and temporally) video frames and then finetunes the TAL head on full-resolution videos. The main contribution of this paper is resolving the task discrepancy problem for the video encoder when training TAL models. Extensive experiments are conducted in the paper to compare the lofi setting with several state-of-the-arts, as well as similar models under different pretraining conditions. Results indicate the proposed method outperforms the baselines.