Low-Fidelity Video Encoder Optimization for Temporal Action Localization

1 Introduction . Video analysis has become an increasingly important area of research , encompassing multiple relevant problems such as action recognition [ 8 , 13 ] , temporal action localization [ 68 , 7 , 12 , 22 , 59 , 60 , 55 , 67 , 15 ] , video grounding [ 36 , 65 , 64 , 66 , 49 , 16 , 24 ] , and video question answering [ 25 , 31 ] . Among those , temporal action localization ( TAL ) [ 68 , 22 ] is a fundamental task , as natural videos are not temporally trimmed . Given an untrimmed video , TAL aims to identify the start and end points of all action instances and recognize their category labels simultaneously . A typical TAL model is based on deep convolutional neural networks ( CNNs ) composed of two modules : a video encoder and a TAL head . The video encoder is often shared across different TAL methods ( e.g. , G-TAD [ 60 ] , BC-GNN [ 3 ] ) ∗Work done during an internship at Samsung AI Centre . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . by taking a specific off-the-shelf action classification model ( e.g. , C3D [ 51 ] , I3D [ 8 ] , TSM [ 33 ] ) , with the differences residing only in the TAL head . However , instead of short ( e.g. , 10 seconds ) trimmed video clips as in action recognition , the input videos to a TAL model are characterized by much longer temporal duration ( e.g. , 120 seconds ) . This causes unique computational challenges that remain unsolved , particularly in model optimization . In standard optimization of a TAL model , a two-stage transfer learning pipeline is often involved : 1 . First , the video encoder is optimized on a large source video classification dataset ( e.g. , Kinetics [ 28 ] ) and , optionally , finetunned on the trimmed version of the target dataset under action classification supervision ; 2 . Second , the video encoder is frozen and the TAL head is optimized on the target action localization dataset ( e.g. , ActivityNet [ 22 ] , HACS [ 68 ] ) under TAL task supervision . With this widely-used TAL training pipeline , the video encoder is only optimal for action classification but not for the target TAL task . Specifically , the video encoder is trained so that different short segments within an action sequence are mapped to similar outputs , thus encouraging insensitivity to the temporal boundaries of actions . This is not desirable for a TAL model . We identify this as a task discrepancy problem . Consequently , the final TAL model could suffer from suboptimal performance . Indeed , jointly optimizing all components of a CNN architecture end-to-end with the target task ’ s supervision is a common practice , e.g. , training models for object detection in static images [ 18 , 46 , 38 ] . Unfortunately , this turns out to be non-trivial for TAL . As mentioned above , model training is severely restricted by the large input size of untrimmed videos and subject to the memory constraint of GPUs . This is why the two-stage optimization pipeline as described above becomes the most common and feasible choice in practice for optimizing a TAL model . On the other hand , existing transfer learning methods mostly focus on tackling the data distribution shift problem across different datasets [ 70 , 50 ] , rather than the task shift problem we study here . Regardless , we believe that solving this limitation of the TAL training design bears a great potential for improving model performance . In this work , we present a simple yet effective low-fidelity ( LoFi ) video encoder optimization method particularly designed for better TAL model training . It is designed to adapt the video encoder from action classification to TAL whilst subject to the same hardware budget . This is achieved by introducing a simple strategy characterized by a new intermediate training stage where both the video encoder and the TAL head are optimized end-to-end using a lower temporal and/or spatial resolution ( i.e. , low-fidelity ) in the mini-batch construction . Compared to the standard training method , our proposed strategy does not increase the GPU memory standard ( often a hard constraint for many practitioners ) . Crucially , with our LoFi training the gradients back-propagate to the video encoder from a temporal action localization loss whilst conditioned on the target TAL head , enabling the learning of a video encoder sensitive to the temporal localization objective . We make the following contributions in this work . ( 1 ) We investigate the limitations of the standard optimization method for TAL models , and consider that the task discrepancy problem hinders the performance of existing TAL models . Despite it being a significant ingredient , video encoder optimization is largely ignored by existing TAL methods , left without systematic investigation . ( 2 ) To improve the training of TAL models , we present a novel , simple , and effective low-fidelity ( LoFi ) video encoder optimization method . It is designed specifically to address the task discrepancy problem with the TAL model ’ s video encoder . ( 3 ) Extensive experiments show that the proposed LoFi optimization method yields new state-of-the-art performance when combined with off-the-shelf TAL models ( e.g. , G-TAD [ 60 ] ) . Critically , our method achieves superior efficiency/accuracy trade-off with clear inference cost advantage and good generalizability to varying-capacity video encoders . 2 Related Work . Temporal action localization ( TAL ) models : TAL models can be grouped by architectural design pattern into two categories , one-stage and two-stage architectures . One-stage methods , either predict temporal action boundaries or generate proposals , and classify them within the same network [ 3 , 9 , 21 , 40 , 34 , 58 , 60 , 62 ] . The latter type , two-stage models , generate sets of action proposals ( e.g. , segments ) [ 6 , 12 , 14 , 23 , 39 ] and then an auxiliary head is used for classification of each proposal into an action class [ 35 , 47 , 48 , 63 , 69 ] . In this work , rather than introducing a novel model design , we focus on the training of generic TAL models , with a particular aim to improve the video encoder optimization . This is a relatively less investigated aspect in the TAL literature . Video encoders in TAL : The video encoder is an indispensable part of a TAL model . Main design choices include the base architecture of video encoder and its optimization procedure . With regards to the architecture , the two-stream Temporal Segment Network ( TSN ) [ 54 ] is one of the most common video encoders in existing TAL methods [ 3 , 34 , 35 , 60 ] . Concretely , these works use two TSN networks , one with a ResNet50 [ 20 ] backbone trained on RGB video frames , and the other with a BN-Inception backbone [ 26 ] trained on optical flow . Other alternatives used as a video encoder for TAL include two-stream I3D model [ 8 ] ( see [ 19 , 63 ] ) and Pseudo-3D [ 45 ] ( see [ 40 ] ) . In terms of optimization , a typical paradigm is two-staged : first pre-training the video encoder and then , in a second stage , training the TAL head of the model with the video encoder fixed . This is constrained by the inherent hardware budget derived from having a large per-video input size . In particular , the video encoder is pre-trained using a cross-entropy loss for action recognition on a large-scale video classification dataset such as Kinetics [ 28 , 64 ] . An optional step is to further pre-train it on the foreground segments of the target TAL dataset [ 34 , 60 , 11 , 44 ] . This brings a mismatch between training and inference for the video encoder , which we call a task discrepancy problem . More specifically , although trained to distinguish the content of different action classes , the video encoder is less sensitive to action temporal boundaries and thus less effective for the TAL task . In fact , due to their inherent design , CNNs have limited localization capabilities [ 41 ] , unless they are augmented with specialized localization-specific layers [ 37 ] . Additionally , the action classification task focuses only on the foreground content whilst ignoring per-class background segments , including the transition between foreground and background . In this paper we propose a novel low-fidelity video encoder pre-training method to solve this limitation with existing TAL methods . While current TAL literature mostly relies on pre-training through supervised learning , the rapid advancement of self-supervised learning makes it a promising alternative [ 2 , 4 , 42 , 43 , 56 ] . Some works have focused on finding effective temporal-related pretext tasks , from frame ordering learnt through triplets of frames [ 43 ] , to sorting the frames of a sequence [ 30 ] , distinguishing whether sequences are played forward or backwards [ 56 ] or through playback speed-related pretext tasks [ 4 , 61 , 53 , 27 ] . These methods exploit video-specific characteristics to force the network to focus on some sort of semantic content within video , inducing representations capturing long-term temporal semantic relations , but force invariance to or ignore the relative positioning of the snippets within the action instances . They are thus not suited for pre-training the video encoder of a TAL model . Very recent works [ 1 , 59 ] have exploited some of the aforementioned techniques for better pre-training of action localization models . For example , localization-tailored data augmentation and classification is adopted by [ 59 ] . However , these works introduce a large amount of extra video data and additional stream networks , both of which are expensive in terms of memory and computation . In contrast , our method aims to improve TAL modelling directly without the need for learning from extra training video data and using an expensive second network , nor relying on optical flow obtained at high computational cost . 3 Method . A TAL model takes as input a long untrimmed video with a varying number of frames . For design convenience , it is typical to represent a varying-length video by decomposing it into a fixed-length sequence of L snippets . The definition of a snippet is the same as in action recognition , where first a number of consecutive frames ( e.g. , 64 ) is selected and then sub-sampled with stride r ( e.g. , stride 8 to obtain 8-frame snippets ) . To represent a snippet , one first applies a video encoder to extract frame-level feature vectors and then averages them to obtain the snippet-level feature representation [ 6 , 12 , 17 , 35 ] . The resulting snippet feature sequence is denoted as X ∈ RC×L , where C is the feature dimension of each snippet , and L is the number of snippets . In the training set , each video is associated with its ground truth , consisting of a set of action instance annotations Ψ . In particular , each action instance is represented as a segment , each including the start time , the end time , and the action class label . The objective is to train a TAL model that can accurately localize all the target action instances in a given untrimmed video . To that end , the model predicts a varying number of action instances Φ , each comprised of the predicted temporal boundaries , action class , and confidence score .

Summary: Models for temporal action localization (TAL) in long, untrimmed video are typically trained in two stages, a video encoder pretrained on a auxiliary video clip classification dataset and then frozen, followed by a TAL head. End-to-end training of the encoder and TAL head is infeasible because the encoder activations for an entire, long video will not fit in GPU RAM. This paper proposes a solution: use the ideas of mulgrid training [51] to do end-to-end training of a lower resolution video encoder and TAL head. They explore strategies for cycling through regimes of lower spatial, temporal, and spatio-temporal resolution training, and demonstrate performance gains on the ActivityNet and HACS datasets, using a number of backbone networks and TAL heads.