Boundary-aware Pre-training for Video Scene Segmentation

1 INTRODUCTION . The video scene segmentation is a task of identifying scene boundaries from a video where a scene is defined as a semantic unit for making a story and is composed of a series of semantically cohesive shots—a set of frames captured by the same camera during an uninterrupted period of time—in the same context . Localizing scene boundaries is a significant step towards the high-level video understanding because dividing a long video into a set of meaningful scenes enables models to inspect the individual incidents from complex story . One of the biggest challenges with temporal semantic segmentation is that it is not achieved simply by detecting changes in visual cues . As shown in Figure 1 ( a ) , we present an example of nine shots , all of which belong to the same scene , where two characters are talking on the phone . We can see that the overall visual cues within the scene do not stay the same but rather change repeatedly when each character appears . On the other hand , as presented in Figure 1 ( b ) , the other example shows two different scenes which contain visually similar shots ( highlighted in blue ) where the same character appears in the same place . Therefore , it is expected that two adjacent scenes which share shots with similar visual cues need to be contextually discriminated . From this observation , it is important for the video scene segmentation task to model contextual relation between shots by maximizing 1 ) intra-scene similarity ( i.e. , the shots belonging to the same scene should be close to each other ) , and 2 ) inter-scene discrimination across two adjacent scenes ( i.e. , two neighbor shots across the scene boundary should be distinguishable ) . Supervised learning approaches ( e.g. , Rao et al . ( 2020 ) ) are clearly limited due to the lack of largescale datasets with reliable ground-truth annotations . Recently , self-supervision ( Chen et al. , 2020a ; Caron et al. , 2020 ; He et al. , 2020 ; Roh et al. , 2021 ) is spotlighted through its effectiveness in learning in-domain representation without relying on costly ground truth annotations . The selfsupervised learning methods ( Chen et al. , 2021 ; Feichtenhofer et al. , 2021 ; Dave et al. , 2021 ; Qian et al. , 2021 ) in the video domain are often designed to learn spatio-temporal patterns in short clips ( e.g. , shots in movies ) . This kind of learned representation is generic and can be applied to many video understanding tasks ( e.g. , action classification ) . However , such representation is not sufficient for video scene segmentation because this task requires not only a good representation for individual shots but also contextual representation considering neighboring shots at a higher level as illustrated in Figure 1 . Motivated by this , we set our main goal to design effective self-supervised objectives ( i.e. , pretext tasks ) that maximize intra-scene similarity as well as discriminate shots from different scenes . For the purpose , this raises a penetrating question : how can we design boundary-relevant pretext tasks without access to ground truth boundary annotations ? We introduce a novel Boundary-aware Self-Supervised Learning ( BaSSL ) framework . Our main idea of BaSSL is to localize a pseudo-boundary , which is obtained by dividing the input sequence of shots into two semantically disjoint sub-sequences , and use it to define pretext tasks that are beneficial to the video scene segmentation task . On top of the discovered two sub-sequences and a pseudo-boundary , three boundary-aware pretext tasks are proposed : 1 ) Shot-Scene Matching ( SSM ) ; 2 ) Contextual Group Matching ( CGM ) ; and 3 ) Pseudo-boundary Prediction ( PP ) . Note that SSM and CGM encourage the model to maximize intra-scene similarity and inter-scene discrimination while PP enables the model to learn the capability of identifying transitional moments . In addition , we perform Masked Shot Modeling ( MSM ) task inspired by Sun et al . ( 2019a ) to further learn temporal relationship between shots . The comprehensive analysis demonstrates the effectiveness of the proposed framework ( i.e. , pre-training of contextual relationship between shots ) as well as the contribution of the proposed individual components ( i.e. , the algorithm for pseudo-boundary discovery and boundary-aware pretext tasks ) . Our main contributions are summarized as follows : ( i ) We introduce a novel boundary-aware pre-training framework which adopts dynamic time warping ( DTW ) algorithm to identify pseudoboundaries and use them as self-supervision to facilitate the pre-training ; ( ii ) we propose three boundary-aware pretext tasks , which are carefully designed to learn essential capabilities required for the video scene segmentation task ; ( iii ) we perform extensive ablations to demonstrate the effectiveness of the proposed framework , including the observation that our framework is complementary to the existing framework ; ( iv ) we achieve the new state-of-the-art on the MovieNet-SSeg benchmark while outperforming existing self-supervised learning-based methods by large margins . 2 RELATED WORK . Video scene segmentation approaches formulate the task as a problem of temporal grouping of shots . In this formulation , the optimal grouping can be achieved by clustering-based ( Rui et al. , 1998 ; Rasheed & Shah , 2003 ; 2005 ; Chasanis et al. , 2008 ) , dynamic programming-based ( Han & Wu , 2011 ; Tapaswi et al. , 2014 ; Rotman et al. , 2017 ) or multi-modal input-based ( Liang et al. , 2009 ; Sidiropoulos et al. , 2011 ) methods . However , the aforementioned methods have been trained and evaluated on small-scale datasets such as OVSD ( Rotman et al. , 2016 ) and BBC ( Baraldi et al. , 2015 ) which can produce a poorly generalized model . Recently , Huang et al . ( 2020 ) introduce a largescale video scene segmentation dataset ( i.e. , MovieNet-SSeg ) that contains hundreds of movies . Training with large-scale data , Rao et al . ( 2020 ) proposes a strong supervised baseline model that performs a shot-level binary classification followed by grouping using the prediction scores . In addition , Chen et al . ( 2021 ) proposes a shot contrastive pre-training method that learns shot-level representation . We found ShotCoL ( Chen et al. , 2021 ) to be the most similar work to our method . However , our method is different from ShotCoL in that we specifically focus on learning contextual representations by considering the relationship between shots . We refer interested readers to the supplementary material for a more detailed analysis of this . Action segmentation in videos is one of the related works for video scene segmentation , which identifies action labels of individual frames , thus divides a video into a series of action segments . Supervised methods ( Lea et al. , 2016 ; Farha & Gall , 2019 ) proposed CNN-based architectures to effectively capture temporal relationship between frames in order to address an over-segmentation issue . As frame-level annotations are prohibitively costly to acquire , weakly supervised methods ( Chang et al. , 2019 ; Li et al. , 2019 ; Li & Todorovic , 2020 ; Souri et al. , 2021 ; Shen et al. , 2021 ; Zhukov et al. , 2019 ; Fried et al. , 2020 ) have been suggested to use an ordered list of actions occurring in a video as supervision . Most of the methods are trained to find ( temporal ) semantic alignment between frames and a given action list using an HMM-based architecture ( Kuehne et al. , 2018 ) , a dynamic programming-based assignment algorithm ( Fried et al. , 2020 ) or a DTW-based temporal alignment method ( Chang et al. , 2019 ) . Recently , unsupervised methods ( Kumar et al. , 2021 ; Wang et al. , 2021 ; Kukleva et al. , 2019 ; Li & Todorovic , 2021 ; VidalMata et al. , 2021 ) have been further proposed ; in a nutshell , clustering-based prototypes are discovered from unlabeled videos , then the methods segment the videos by assigning prototypes ( corresponding to one of the actions ) into frames . In contrast to the action segmentation task that is limited to localizing segments each of which represents a single action within an activity , video scene segmentation requires localizing more complex segments each of which may be composed of more than two actions ( or activities ) . Self-supervised learning in videos has been actively studied for the recent years with approaches proposing various pretext tasks such as future frame prediction ( Srivastava et al. , 2015 ; Vondrick et al. , 2016 ; Ahsan et al. , 2018 ) , temporal ordering of frames ( Misra et al. , 2016 ; Lee et al. , 2017 ; Xu et al. , 2019 ) , geometric transformations prediction ( Jing & Tian , 2018 ) , colorization of videos ( Vondrick et al. , 2018 ) and contrastive prediction ( Feichtenhofer et al. , 2021 ; Qian et al. , 2021 ; Dave et al. , 2021 ) . In addition , CBT ( Sun et al. , 2019a ; b ) proposes a pretext task of masked frame modeling to learn temporal dependency between frames ( or clips ) . Note that since most of those methods are proposed for the classification task , they would be sub-optimal to the video scene segmentation task . On the other hand , BSP ( Xu et al. , 2020 ) proposes boundary-sensitive pre-text tasks based on synthesized pseudo-boundaries that are obtained by concatenating two clips sampled from different videos . However , strictly speaking , BSP is not a self-supervised learning algorithm since it requires video-level class labels to synthesize pseudo-boundaries ; the proposed pretext tasks are not applicable to videos such as movies that are hard to define semantic labels . Also , note that we empirically show that pseudo-boundaries identified by our method are more effective for pre-training than synthesized pseudo-boundaries . 3 BOUNDARY-AWARE SELF-SUPERVISED LEARNING ( BASSL ) . In this section , we introduce our proposed approach , Boundary-aware Self-Supervised Learning ( BaSSL ) . We start with the problem formulation followed by the model overview . Then , we describe our novel boundary-aware pretext tasks for pre-training . 3.1 PROBLEM FORMULATION . Terminologies A video ( e.g. , documentaries , TV episodes and movies ) is a sequence of scenes , defined as a semantic unit for making a story . A scene is a series of shots , which is a set of frames physically captured by the same camera during an uninterrupted period of time . Problem Definition Given a video , which contains a series of N shots { s1 , ... , sN } with class labels { y1 , ... , yN } where yi ∈ { 0 , 1 } indicating if it is at the scene boundary ( more precisely , if it is the last shot of a scene ) , the video scene segmentation task is formulated as a simple binary classification problem at individual shot level . By definition , a scene boundary is where the semantic of a shot is considerably different from its ( one-way ) neighbors . Thus , it is in nature important to capture and leverage contextual transition across the scenes . Consequently , it is a common practice that the information of the neighbor shots are leveraged together when determining scene boundaries . With this formulation , existing supervised learning approaches typically train a parameterized ( θ ) model by maximizing the expected log-likelihood : Pre-training Stage Fine-tuning Stage Pseudo-boundary Shot θ∗ = arg max θ E [ log pθ ( yn|Sn ) ] , ( 1 ) where Sn = { sn−K , ... , sn , ... , sn+K } is a set of 2K + 1 shots centered at nth shot sn , and K is the number of neighbor shots before and after sn . Note that each shot s is given by a set of Nk keyframes , resulting in a tensor with size of ( Nk , C , H , W ) where C , H and W are the RGB channels , the frame height and the frame width , respectively .

Summary: The paper introduces a boundary-aware self-supervised learning framework for video scene segmentation. Based on the contrastive learning protocol, the authors employ DTW to generate dimidiate video segmentation from unlabeled video. Using these segments, the authors employ four loss functions to pre-train a backbone network.