Boundary-aware Pre-training for Video Scene Segmentation

1 INTRODUCTION . The video scene segmentation is a task of identifying scene boundaries from a video where a scene is defined as a semantic unit for making a story and is composed of a series of semantically cohesive shots—a set of frames captured by the same camera during an uninterrupted period of time—in the same context . Localizing scene boundaries is a significant step towards the high-level video understanding because dividing a long video into a set of meaningful scenes enables models to inspect the individual incidents from complex story . One of the biggest challenges with temporal semantic segmentation is that it is not achieved simply by detecting changes in visual cues . As shown in Figure 1 ( a ) , we present an example of nine shots , all of which belong to the same scene , where two characters are talking on the phone . We can see that the overall visual cues within the scene do not stay the same but rather change repeatedly when each character appears . On the other hand , as presented in Figure 1 ( b ) , the other example shows two different scenes which contain visually similar shots ( highlighted in blue ) where the same character appears in the same place . Therefore , it is expected that two adjacent scenes which share shots with similar visual cues need to be contextually discriminated . From this observation , it is important for the video scene segmentation task to model contextual relation between shots by maximizing 1 ) intra-scene similarity ( i.e. , the shots belonging to the same scene should be close to each other ) , and 2 ) inter-scene discrimination across two adjacent scenes ( i.e. , two neighbor shots across the scene boundary should be distinguishable ) . Supervised learning approaches ( e.g. , Rao et al . ( 2020 ) ) are clearly limited due to the lack of largescale datasets with reliable ground-truth annotations . Recently , self-supervision ( Chen et al. , 2020a ; Caron et al. , 2020 ; He et al. , 2020 ; Roh et al. , 2021 ) is spotlighted through its effectiveness in learning in-domain representation without relying on costly ground truth annotations . The selfsupervised learning methods ( Chen et al. , 2021 ; Feichtenhofer et al. , 2021 ; Dave et al. , 2021 ; Qian et al. , 2021 ) in the video domain are often designed to learn spatio-temporal patterns in short clips ( e.g. , shots in movies ) . This kind of learned representation is generic and can be applied to many video understanding tasks ( e.g. , action classification ) . However , such representation is not sufficient for video scene segmentation because this task requires not only a good representation for individual shots but also contextual representation considering neighboring shots at a higher level as illustrated in Figure 1 . Motivated by this , we set our main goal to design effective self-supervised objectives ( i.e. , pretext tasks ) that maximize intra-scene similarity as well as discriminate shots from different scenes . For the purpose , this raises a penetrating question : how can we design boundary-relevant pretext tasks without access to ground truth boundary annotations ? We introduce a novel Boundary-aware Self-Supervised Learning ( BaSSL ) framework . Our main idea of BaSSL is to localize a pseudo-boundary , which is obtained by dividing the input sequence of shots into two semantically disjoint sub-sequences , and use it to define pretext tasks that are beneficial to the video scene segmentation task . On top of the discovered two sub-sequences and a pseudo-boundary , three boundary-aware pretext tasks are proposed : 1 ) Shot-Scene Matching ( SSM ) ; 2 ) Contextual Group Matching ( CGM ) ; and 3 ) Pseudo-boundary Prediction ( PP ) . Note that SSM and CGM encourage the model to maximize intra-scene similarity and inter-scene discrimination while PP enables the model to learn the capability of identifying transitional moments . In addition , we perform Masked Shot Modeling ( MSM ) task inspired by Sun et al . ( 2019a ) to further learn temporal relationship between shots . The comprehensive analysis demonstrates the effectiveness of the proposed framework ( i.e. , pre-training of contextual relationship between shots ) as well as the contribution of the proposed individual components ( i.e. , the algorithm for pseudo-boundary discovery and boundary-aware pretext tasks ) . Our main contributions are summarized as follows : ( i ) We introduce a novel boundary-aware pre-training framework which adopts dynamic time warping ( DTW ) algorithm to identify pseudoboundaries and use them as self-supervision to facilitate the pre-training ; ( ii ) we propose three boundary-aware pretext tasks , which are carefully designed to learn essential capabilities required for the video scene segmentation task ; ( iii ) we perform extensive ablations to demonstrate the effectiveness of the proposed framework , including the observation that our framework is complementary to the existing framework ; ( iv ) we achieve the new state-of-the-art on the MovieNet-SSeg benchmark while outperforming existing self-supervised learning-based methods by large margins . 2 RELATED WORK . Video scene segmentation approaches formulate the task as a problem of temporal grouping of shots . In this formulation , the optimal grouping can be achieved by clustering-based ( Rui et al. , 1998 ; Rasheed & Shah , 2003 ; 2005 ; Chasanis et al. , 2008 ) , dynamic programming-based ( Han & Wu , 2011 ; Tapaswi et al. , 2014 ; Rotman et al. , 2017 ) or multi-modal input-based ( Liang et al. , 2009 ; Sidiropoulos et al. , 2011 ) methods . However , the aforementioned methods have been trained and evaluated on small-scale datasets such as OVSD ( Rotman et al. , 2016 ) and BBC ( Baraldi et al. , 2015 ) which can produce a poorly generalized model . Recently , Huang et al . ( 2020 ) introduce a largescale video scene segmentation dataset ( i.e. , MovieNet-SSeg ) that contains hundreds of movies . Training with large-scale data , Rao et al . ( 2020 ) proposes a strong supervised baseline model that performs a shot-level binary classification followed by grouping using the prediction scores . In addition , Chen et al . ( 2021 ) proposes a shot contrastive pre-training method that learns shot-level representation . We found ShotCoL ( Chen et al. , 2021 ) to be the most similar work to our method . However , our method is different from ShotCoL in that we specifically focus on learning contextual representations by considering the relationship between shots . We refer interested readers to the supplementary material for a more detailed analysis of this . Action segmentation in videos is one of the related works for video scene segmentation , which identifies action labels of individual frames , thus divides a video into a series of action segments . Supervised methods ( Lea et al. , 2016 ; Farha & Gall , 2019 ) proposed CNN-based architectures to effectively capture temporal relationship between frames in order to address an over-segmentation issue . As frame-level annotations are prohibitively costly to acquire , weakly supervised methods ( Chang et al. , 2019 ; Li et al. , 2019 ; Li & Todorovic , 2020 ; Souri et al. , 2021 ; Shen et al. , 2021 ; Zhukov et al. , 2019 ; Fried et al. , 2020 ) have been suggested to use an ordered list of actions occurring in a video as supervision . Most of the methods are trained to find ( temporal ) semantic alignment between frames and a given action list using an HMM-based architecture ( Kuehne et al. , 2018 ) , a dynamic programming-based assignment algorithm ( Fried et al. , 2020 ) or a DTW-based temporal alignment method ( Chang et al. , 2019 ) . Recently , unsupervised methods ( Kumar et al. , 2021 ; Wang et al. , 2021 ; Kukleva et al. , 2019 ; Li & Todorovic , 2021 ; VidalMata et al. , 2021 ) have been further proposed ; in a nutshell , clustering-based prototypes are discovered from unlabeled videos , then the methods segment the videos by assigning prototypes ( corresponding to one of the actions ) into frames . In contrast to the action segmentation task that is limited to localizing segments each of which represents a single action within an activity , video scene segmentation requires localizing more complex segments each of which may be composed of more than two actions ( or activities ) . Self-supervised learning in videos has been actively studied for the recent years with approaches proposing various pretext tasks such as future frame prediction ( Srivastava et al. , 2015 ; Vondrick et al. , 2016 ; Ahsan et al. , 2018 ) , temporal ordering of frames ( Misra et al. , 2016 ; Lee et al. , 2017 ; Xu et al. , 2019 ) , geometric transformations prediction ( Jing & Tian , 2018 ) , colorization of videos ( Vondrick et al. , 2018 ) and contrastive prediction ( Feichtenhofer et al. , 2021 ; Qian et al. , 2021 ; Dave et al. , 2021 ) . In addition , CBT ( Sun et al. , 2019a ; b ) proposes a pretext task of masked frame modeling to learn temporal dependency between frames ( or clips ) . Note that since most of those methods are proposed for the classification task , they would be sub-optimal to the video scene segmentation task . On the other hand , BSP ( Xu et al. , 2020 ) proposes boundary-sensitive pre-text tasks based on synthesized pseudo-boundaries that are obtained by concatenating two clips sampled from different videos . However , strictly speaking , BSP is not a self-supervised learning algorithm since it requires video-level class labels to synthesize pseudo-boundaries ; the proposed pretext tasks are not applicable to videos such as movies that are hard to define semantic labels . Also , note that we empirically show that pseudo-boundaries identified by our method are more effective for pre-training than synthesized pseudo-boundaries . 3 BOUNDARY-AWARE SELF-SUPERVISED LEARNING ( BASSL ) . In this section , we introduce our proposed approach , Boundary-aware Self-Supervised Learning ( BaSSL ) . We start with the problem formulation followed by the model overview . Then , we describe our novel boundary-aware pretext tasks for pre-training . 3.1 PROBLEM FORMULATION . Terminologies A video ( e.g. , documentaries , TV episodes and movies ) is a sequence of scenes , defined as a semantic unit for making a story . A scene is a series of shots , which is a set of frames physically captured by the same camera during an uninterrupted period of time . Problem Definition Given a video , which contains a series of N shots { s1 , ... , sN } with class labels { y1 , ... , yN } where yi ∈ { 0 , 1 } indicating if it is at the scene boundary ( more precisely , if it is the last shot of a scene ) , the video scene segmentation task is formulated as a simple binary classification problem at individual shot level . By definition , a scene boundary is where the semantic of a shot is considerably different from its ( one-way ) neighbors . Thus , it is in nature important to capture and leverage contextual transition across the scenes . Consequently , it is a common practice that the information of the neighbor shots are leveraged together when determining scene boundaries . With this formulation , existing supervised learning approaches typically train a parameterized ( θ ) model by maximizing the expected log-likelihood : Pre-training Stage Fine-tuning Stage Pseudo-boundary Shot θ∗ = arg max θ E [ log pθ ( yn|Sn ) ] , ( 1 ) where Sn = { sn−K , ... , sn , ... , sn+K } is a set of 2K + 1 shots centered at nth shot sn , and K is the number of neighbor shots before and after sn . Note that each shot s is given by a set of Nk keyframes , resulting in a tensor with size of ( Nk , C , H , W ) where C , H and W are the RGB channels , the frame height and the frame width , respectively .

Summary: This paper introduces a self-supervised learning method for the video scene segmentation task, which needs to temporally localize scene boundaries in a video. Their proposed method defines pseudo-boundary from a sequence of shots by splitting it into two sub-sequences and train a model to predict the pseudo-boundary with three novel tasks: Shot-Scene Matching (SSM), Contextual Group Matching (CGM), and Pseudo-boundary Prediction (PP). Experiments on pre-training and transfer learning show that this method is critical to improving the video scene segmentation performance, and it achieves the new state-of-the-art on the MovieNet-SSeg benchmark.