Year: 2021

Abstract:
Self-supervised learning has drawn attention through its effectiveness in learning in-domain representations with no ground-truth annotations; in particular, it is shown that properly designed pretext tasks (e.g., contrastive prediction task) bring significant performance gains for a downstream task (e.g., classification task). Inspired from this, we tackle video scene segmentation, which is a task of temporally localizing scene boundaries in a video, with a self-supervised learning framework where we mainly focus on designing effective pretext tasks. In our framework, we discover a pseudo-boundary from a sequence of shots by splitting it into two continuous, non-overlapping sub-sequences and leverage the pseudo-boundary to facilitate the pre-training. Based on this, we introduce three novel boundary-aware pretext tasks: 1) Shot-Scene Matching (SSM), 2) Contextual Group Matching (CGM) and 3) Pseudo-boundary Prediction (PP); SSM and CGM guide the model to maximize intra-scene similarity and inter-scene discrimination while PP encourages the model to identify transitional moments. Through comprehensive analysis, we empirically show that pre-training and transferring contextual representation are both critical to improving the video scene segmentation performance. Lastly, we achieve the new state-of-the-art on the MovieNet-SSeg benchmark. The code will be released.

heading: 1 INTRODUCTION
text: The video scene segmentation is a task of identifying scene boundaries from a video where a scene is defined as a semantic unit for making a story and is composed of a series of semantically cohesive shots—a set of frames captured by the same camera during an uninterrupted period of time—in the same context. Localizing scene boundaries is a significant step towards the high-level video understanding because dividing a long video into a set of meaningful scenes enables models to inspect the individual incidents from complex story.
One of the biggest challenges with temporal semantic segmentation is that it is not achieved simply by detecting changes in visual cues. As shown in Figure 1(a), we present an example of nine shots, all of which belong to the same scene, where two characters are talking on the phone. We can see that the overall visual cues within the scene do not stay the same but rather change repeatedly when each character appears. On the other hand, as presented in Figure 1(b), the other example shows two different scenes which contain visually similar shots (highlighted in blue) where the same character appears in the same place. Therefore, it is expected that two adjacent scenes which share shots with similar visual cues need to be contextually discriminated. From this observation, it is important for the video scene segmentation task to model contextual relation between shots by maximizing 1) intra-scene similarity (i.e., the shots belonging to the same scene should be close to each other), and 2) inter-scene discrimination across two adjacent scenes (i.e., two neighbor shots across the scene boundary should be distinguishable).
Supervised learning approaches (e.g., Rao et al. (2020)) are clearly limited due to the lack of largescale datasets with reliable ground-truth annotations. Recently, self-supervision (Chen et al., 2020a; Caron et al., 2020; He et al., 2020; Roh et al., 2021) is spotlighted through its effectiveness in learning in-domain representation without relying on costly ground truth annotations. The selfsupervised learning methods (Chen et al., 2021; Feichtenhofer et al., 2021; Dave et al., 2021; Qian et al., 2021) in the video domain are often designed to learn spatio-temporal patterns in short clips
(e.g., shots in movies). This kind of learned representation is generic and can be applied to many video understanding tasks (e.g., action classification). However, such representation is not sufficient for video scene segmentation because this task requires not only a good representation for individual shots but also contextual representation considering neighboring shots at a higher level as illustrated in Figure 1. Motivated by this, we set our main goal to design effective self-supervised objectives (i.e., pretext tasks) that maximize intra-scene similarity as well as discriminate shots from different scenes. For the purpose, this raises a penetrating question: how can we design boundary-relevant pretext tasks without access to ground truth boundary annotations?
We introduce a novel Boundary-aware Self-Supervised Learning (BaSSL) framework. Our main idea of BaSSL is to localize a pseudo-boundary, which is obtained by dividing the input sequence of shots into two semantically disjoint sub-sequences, and use it to define pretext tasks that are beneficial to the video scene segmentation task. On top of the discovered two sub-sequences and a pseudo-boundary, three boundary-aware pretext tasks are proposed: 1) Shot-Scene Matching (SSM); 2) Contextual Group Matching (CGM); and 3) Pseudo-boundary Prediction (PP). Note that SSM and CGM encourage the model to maximize intra-scene similarity and inter-scene discrimination while PP enables the model to learn the capability of identifying transitional moments. In addition, we perform Masked Shot Modeling (MSM) task inspired by Sun et al. (2019a) to further learn temporal relationship between shots. The comprehensive analysis demonstrates the effectiveness of the proposed framework (i.e., pre-training of contextual relationship between shots) as well as the contribution of the proposed individual components (i.e., the algorithm for pseudo-boundary discovery and boundary-aware pretext tasks).
Our main contributions are summarized as follows: (i) We introduce a novel boundary-aware pre-training framework which adopts dynamic time warping (DTW) algorithm to identify pseudoboundaries and use them as self-supervision to facilitate the pre-training; (ii) we propose three boundary-aware pretext tasks, which are carefully designed to learn essential capabilities required for the video scene segmentation task; (iii) we perform extensive ablations to demonstrate the effectiveness of the proposed framework, including the observation that our framework is complementary to the existing framework; (iv) we achieve the new state-of-the-art on the MovieNet-SSeg benchmark while outperforming existing self-supervised learning-based methods by large margins.

heading: 2 RELATED WORK
text: Video scene segmentation approaches formulate the task as a problem of temporal grouping of shots. In this formulation, the optimal grouping can be achieved by clustering-based (Rui et al., 1998; Rasheed & Shah, 2003; 2005; Chasanis et al., 2008), dynamic programming-based (Han & Wu, 2011; Tapaswi et al., 2014; Rotman et al., 2017) or multi-modal input-based (Liang et al., 2009; Sidiropoulos et al., 2011) methods. However, the aforementioned methods have been trained and evaluated on small-scale datasets such as OVSD (Rotman et al., 2016) and BBC (Baraldi et al., 2015) which can produce a poorly generalized model. Recently, Huang et al. (2020) introduce a largescale video scene segmentation dataset (i.e., MovieNet-SSeg) that contains hundreds of movies. Training with large-scale data, Rao et al. (2020) proposes a strong supervised baseline model that performs a shot-level binary classification followed by grouping using the prediction scores. In addition, Chen et al. (2021) proposes a shot contrastive pre-training method that learns shot-level representation. We found ShotCoL (Chen et al., 2021) to be the most similar work to our method. However, our method is different from ShotCoL in that we specifically focus on learning contextual
representations by considering the relationship between shots. We refer interested readers to the supplementary material for a more detailed analysis of this.
Action segmentation in videos is one of the related works for video scene segmentation, which identifies action labels of individual frames, thus divides a video into a series of action segments. Supervised methods (Lea et al., 2016; Farha & Gall, 2019) proposed CNN-based architectures to effectively capture temporal relationship between frames in order to address an over-segmentation issue. As frame-level annotations are prohibitively costly to acquire, weakly supervised methods (Chang et al., 2019; Li et al., 2019; Li & Todorovic, 2020; Souri et al., 2021; Shen et al., 2021; Zhukov et al., 2019; Fried et al., 2020) have been suggested to use an ordered list of actions occurring in a video as supervision. Most of the methods are trained to find (temporal) semantic alignment between frames and a given action list using an HMM-based architecture (Kuehne et al., 2018), a dynamic programming-based assignment algorithm (Fried et al., 2020) or a DTW-based temporal alignment method (Chang et al., 2019). Recently, unsupervised methods (Kumar et al., 2021; Wang et al., 2021; Kukleva et al., 2019; Li & Todorovic, 2021; VidalMata et al., 2021) have been further proposed; in a nutshell, clustering-based prototypes are discovered from unlabeled videos, then the methods segment the videos by assigning prototypes (corresponding to one of the actions) into frames. In contrast to the action segmentation task that is limited to localizing segments each of which represents a single action within an activity, video scene segmentation requires localizing more complex segments each of which may be composed of more than two actions (or activities).
Self-supervised learning in videos has been actively studied for the recent years with approaches proposing various pretext tasks such as future frame prediction (Srivastava et al., 2015; Vondrick et al., 2016; Ahsan et al., 2018), temporal ordering of frames (Misra et al., 2016; Lee et al., 2017; Xu et al., 2019), geometric transformations prediction (Jing & Tian, 2018), colorization of videos (Vondrick et al., 2018) and contrastive prediction (Feichtenhofer et al., 2021; Qian et al., 2021; Dave et al., 2021). In addition, CBT (Sun et al., 2019a;b) proposes a pretext task of masked frame modeling to learn temporal dependency between frames (or clips). Note that since most of those methods are proposed for the classification task, they would be sub-optimal to the video scene segmentation task. On the other hand, BSP (Xu et al., 2020) proposes boundary-sensitive pre-text tasks based on synthesized pseudo-boundaries that are obtained by concatenating two clips sampled from different videos. However, strictly speaking, BSP is not a self-supervised learning algorithm since it requires video-level class labels to synthesize pseudo-boundaries; the proposed pretext tasks are not applicable to videos such as movies that are hard to define semantic labels. Also, note that we empirically show that pseudo-boundaries identified by our method are more effective for pre-training than synthesized pseudo-boundaries.

heading: 3 BOUNDARY-AWARE SELF-SUPERVISED LEARNING (BASSL)
text: In this section, we introduce our proposed approach, Boundary-aware Self-Supervised Learning (BaSSL). We start with the problem formulation followed by the model overview. Then, we describe our novel boundary-aware pretext tasks for pre-training.

heading: 3.1 PROBLEM FORMULATION
text: Terminologies A video (e.g., documentaries, TV episodes and movies) is a sequence of scenes, defined as a semantic unit for making a story. A scene is a series of shots, which is a set of frames physically captured by the same camera during an uninterrupted period of time.
Problem Definition Given a video, which contains a series of N shots {s1, ..., sN} with class labels {y1, ..., yN} where yi ∈ {0, 1} indicating if it is at the scene boundary (more precisely, if it is the last shot of a scene), the video scene segmentation task is formulated as a simple binary classification problem at individual shot level. By definition, a scene boundary is where the semantic of a shot is considerably different from its (one-way) neighbors. Thus, it is in nature important to capture and leverage contextual transition across the scenes. Consequently, it is a common practice that the information of the neighbor shots are leveraged together when determining scene boundaries. With this formulation, existing supervised learning approaches typically train a parameterized (θ) model by maximizing the expected log-likelihood:
Pre-training Stage Fine-tuning Stage
Pseudo-boundary Shot
θ∗ = arg max θ E [log pθ(yn|Sn)] , (1)
where Sn = {sn−K , ..., sn, ..., sn+K} is a set of 2K + 1 shots centered at nth shot sn, and K is the number of neighbor shots before and after sn. Note that each shot s is given by a set of Nk keyframes, resulting in a tensor with size of (Nk, C,H,W ) where C, H and W are the RGB channels, the frame height and the frame width, respectively.

heading: 3.2 MODEL OVERVIEW
text: Our method is based on two-stage training following common practice (Chen et al., 2021): pretraining on large-scale unlabeled data with self-supervision and fine-tuning on relatively small labeled data via transfer learning. Our main focus is in the pre-training stage, aiming at designing effective pretext tasks for video scene segmentation.
As illustrated in Figure 2, our model consists of two main components: 1) shot encoder embedding a shot by capturing its spatio-temporal patterns, and 2) contextual relation network (CRN) capturing relationship between shots. Given a sequence Sn = {sn−K , ..., sn, ..., sn+K} of shots centered at sn, two-level representations are extracted as follows:
en = fENC(sn) and Cn = fCRN(En), (2)
where fENC : RNk×C×H×W → RDe and fCRN : R(2K+1)×De→ R(2K+1)×Dc represent the shot encoder and the contextual relation network while De and Dc mean dimensions of encoded and contextualized features, respectively. en is an encoding of shot sn by fENC while En = {en−K , ..., en, ..., en+K} and Cn = {cn−K , ..., cn, ..., cn+K} correspond to the input and output feature sequence for fCRN, respectively.
On top of encoded shot representations, BaSSL extracts a pseudo-boundary (left box of Figure 2) to self-supervise the model instead of relying on ground-truth annotations. To be specific, we leverage the dynamic time warping technique to divide the input sequence of shots into two semantically disjoint sub-sequences and output a pseudo-boundary. (See Section 3.3 for more details.)
Then, as presented in Figure 3, using the discovered pseudo-boundary, we devise three novel boundary-aware pretext tasks in Section 3.4: 1) Shot-Scene Matching to match shots with their associated scenes, 2) Contextual Group Matching to align shots whether they belong to the same scene or not and 3) Pseudo-boundary Prediction to capture semantic changes. In addition, we adopt the masked shot modeling in CBT (Sun et al., 2019a) to further learn temporal relationship between shots. After trained with the four pretext tasks, the model is fine-tuned with labeled video scene segmentation data. (See Section 3.5 for more details.)

heading: 3.3 PSEUDO-BOUNDARY DISCOVERY
text: During the pre-training, no scene boundary annotation is available for the input sequence Sn of shots. Thus, we infer pseudo-boundaries which serve as if they are ground truth labels for boundaryrelevant pretext tasks. In addition, we simplify the problem to always extract a single boundary given an input sequence. This is reasonable because even a sequence of shots without strong scenelevel semantic transition, there always exists a shot within the sequence across which the semantic transition is maximum, and we use this shot as a pseudo-boundary. To support our claim, we show examples in Figure 4 where we intentionally infer a pseudo-boundary based on a sequence of shots sampled from the same scene (that is, no scene boundary exists between shots according to the ground truth); we observe that the resulting two sub-sequences are cognitively distinguishable.
This simplified problem, dividing the input sequence Sn into two continuous, non-overlapping subsequences Sleftn and S right n , can be seen as a temporal alignment problem between Sn and Sslown ; specifically, observing the first shot should belong to Sleftn and the last one to S right n , we define Sslown = {sn−K , sn+K}, which can be seen as a same video with Sn with lower sampling frequency. Finally, the task becomes aligning intermediate shots either to Sleftn or S right n preserving continuity.
Under the problem setting, we adopt dynamic time warping (DTW) (Berndt & Clifford, 1994) to find the optimal temporal alignment between Sn and Sslown . In detail, DTW solves the following optimization problem using dynamic programming to maximize semantic coherence of the resulting two sub-sequences among all possible boundary candidates:
b∗ = arg max b=−K+1,...,K−1
1
b+K b∑ i=−K+1 sim(en−K , en+i) + 1 K − b− 1 K−1∑ j=b+1 sim(en+K , en+j),
(3)
where b is the candidate boundary offset, b∗ is the optimal boundary offset, and sim(x,y) = x >y ‖x‖‖y‖ computes cosine similarity between encodings of the given two shots. Two sub-sequences are inferred as Sleftn = {sn−K , ..., sn+b∗} and S right n = {sn+b∗+1, ..., sn+K}. sn+b∗ is the pseudoboundary shot, which is the last shot of Sleftn . More examples of pseudo-boundaries identified by our algorithm is presented in Figure 7 of the supplementary material. The results are used for learning boundary-aware pretext tasks, which will be described below.

heading: 3.4 PRE-TRAINING OBJECTIVES
text: As illustrated in Figure 3, we adopt four pretext tasks for pre-training—1) shot-scene matching, 2) contextual group matching, 3) pseudo-boundary prediction, and 4) masked shot modeling—and the final pre-training loss is defined by
Lpretrain = LSSM + LCGM + LPP + LMSM. (4)
Shot-Scene Matching (SSM) The objective of this task is to make the representations of a shot and its associated scene similar to each other, while the representations of the shot and other scenes dissimilar. In other words, SSM encourages the model to maximize intra-scene similarity, while
minimizing inter-scene similarity. Considering the splitted two sub-sequences (Sleftn and S right n ) as pseudo-scenes, we train the model using the InfoNCE loss (Oord et al., 2018): LSSM = LNCE ( hSSM(en−K), hSSM(r(S left n )) ) + LNCE ( hSSM(en+K), hSSM(r(S right n )) ) , (5)
LNCE(e, r) = − log exp(sim(e, r)/τ) exp(sim(e, r)/τ) + ∑ ē∈Ne exp(sim(ē, r)/τ) + ∑ r̄∈Nr exp(sim(e, r̄)/τ) ,
(6) where hSSM is a SSM head of a linear layer, τ is a temperature hyperparameter and r(S) means a scene-level representation; we use the averaged encoding of shots in the sub-sequence S. Ne and Nr in Eq. (6) are constructed using other shots and pseudo-scenes in a mini-batch, respectively.
Contextual Group Matching (CGM) Since directly matching representations of shots and scenes would not be effective when the scenes are composed of visually dissimilar shots, CGM is introduced to bridge this gap. Similar to SSM, CGM is also designed to maximize intra-scene similarity and inter-scene discrimination. However, CGM measures semantic coherence of the shots rather than comparing visual cues. With CGM, the model learns to decide if the given two shots belong to the same group (i.e., scene) or not. In detail, we use the center shot sn in the input sequence as the anchor and construct a triplet of (sn, spos, sneg). We sample each shot from Sleftn and S right n ; the one sampled within the same sub-sequence with sn is used as the positive shot spos, while the other as the negative sneg. The CGM loss is defined using a binary cross-entropy loss as follows:
LCGM = − log (hCGM(cn, cpos))− log (1− hCGM(cn, cneg)) , (7) where hCGM is a CGM head taking two shots as input and predicting a matching score. cn, cpos and cneg are the contextualized features by fCRN for the center, positive and negative shots, respectively.
Pseudo-boundary Prediction (PP) Through the above two pretext tasks, our model learns the contextual relationship between shots. In addition to these, we design an extra pretext task, PP, which is more directly related to boundary detection; PP makes the model have a capability of identifying transitional moments that semantic changes. Based on the pseudo-boundary shot and one randomly sampled non-boundary shot, the PP loss is defined as a binary cross-entropy loss:
LPP = − log (hPP(cn+b∗))− log (1− hPP(cb̄)) , (8) where hPP is a PP head that projects the contextualized shot representation to a probability distribution over binary class. cn+b∗ and cb̄ indicate the contextualized representation from fCRN for the pseudo-boundary shot sn+b∗ and randomly sampled non-boundary shot sb̄, respectively.
Masked Shot Modeling (MSM) Inspired by masked frame modeling (Sun et al., 2019a;b), we adopt the MSM task whose goal is to reconstruct the representation of masked shots based on the their surrounding shots. In this task, given a set of encoded shot representations, we randomly apply masking each of them with a probability of 15%. For a setM of masked shot offsets, we learn to regress the output on each masked shot to its encoded shot representation, which is given by
LMSM = ∑ m∈M ‖em − hMSM(cm)‖22, (9)
where hMSM is a MSM head to match the dimension of contextualized shot representation with that of encoded one. em and cm denote the encoded representation by fENC and contextualized representation by fCRN for a masked shot sm, respectively.

heading: 3.5 FINE-TUNING FOR SCENE BOUNDARY DETECTION
text: Recall that we formulate the video scene segmentation as a binary classification task to identify contextual transition across the scene. Different from the pre-training stage, given an input sequence of shots Sn, we employ a scene boundary detection head hSBD to infer a prediction from the contextualized representation (cn) for the center shot sn. Following Chen et al. (2021), we freeze the parameters of the shot encoder and then train only the contextual relation network and the scene boundary detection head using a binary cross-entropy loss with the ground truth label yn as follows:
Lfinetune = −yn log(hSBD(cn)) + (1− yn) log(1− hsbd(cn)). (10) Note that individual shots are decided to be a scene boundary when its prediction score is higher than a pre-defined threshold (set to 0.5).

heading: 4 EXPERIMENT

heading: 4.1 EXPERIMENTAL SETTINGS
text: Dataset We evaluate our proposed method on the MovieNet-SSeg dataset (Huang et al., 2020) that is a sub-dataset of MovieNet, containing 1,100 movies with 1.6M shots. Note that only 318 out of 1,100 movies have scene boundary annotations, which are divided into 190, 64, and 64 movies for training, validation, and test split, respectively. Following Chen et al. (2021), we use the entire 1,100 movies with no ground truth labels for the pre-training and fine-tune the model on the training split. The performance is measured on the test split.
Metric Following Huang et al. (2020), we compare algorithms using Average Precision (AP) and mIoU that measures the averaged intersection over union (IoU) between predicted scene segments and their closest ground truth scene segments. Also, we adopt F1 score and AUC-ROC as additional evaluation metrics. Note that contrary to the previous works (Rao et al., 2020; Chen et al., 2021) that report recall, we use F1 score to consider for balanced comparison between precision and recall. In addition, we report Meta-Sum metric inspired by the works (Chen et al., 2020b; Li et al., 2021) for easy and straightforward comparison of algorithms.
Implementation details We employ ResNet-50 (He et al., 2016) and Transformer (Vaswani et al., 2017) networks as the shot encoder and the contextual relation network, respectively. For both pre-training and fine-tuning stages, we cross-validate the number of neighbor shots among K = {4, 8, 12, 16} and K = 8 is selected due to its good performance and computational efficiency. In all experiments, given a pre-trained model, we fine-tune the model 5 times with different random seeds and report their average score and standard deviation. Note that more details are presented in supplementary material.

heading: 4.2 COMPARISON WITH STATE-OF-THE-ART METHODS
text: We compare our method, BaSSL, with 1) supervised methods including Siamese (Baraldi et al., 2015), MS-LSTM (Huang et al., 2020) and LGSS (Rao et al., 2020), 2) unsupervised methods including GraphCut (Rasheed & Shah, 2005), SCSA (Chasanis et al., 2008), DP (Han & Wu, 2011), StoryGraph (Tapaswi et al., 2014) and Grouping (Rotman et al., 2017), and 3) self-supervised methods including ShotCoL (Chen et al., 2021). Without fine-tuning on the downstream task, BaSSL can be seen as an unsupervised model in that it is trained to predict the pseudo-boundary by the PP task. Table 1 summarizes comparison against state-of-the-art methods. BaSSL without fine-tuning shows competitive or outperforming performance based only on basic visual features compared to competing unsupervised methods. Furthermore, fine-tuning BaSSL with ground-truth scene bound-
aries, AP is improved by 24.71%p and BaSSL outperforms all other algorithms. Finally, through longer pre-training (40 epochs), BaSSL surpasses the state-of-the-art method (i.e., ShotCoL) by a large margin (4.00%p in AP).

heading: 4.3 COMPARISON WITH PRE-TRAINING BASELINES
text: We perform extensive experiments to compare BaSSL with the other pre-training baselines that learn shot-level representation by fENC. In the experiments, we compare the following three types of pretraining approaches; The first group (M1-2) trains fENC using image-level supervision with object labels on ImageNet (Deng et al., 2009) or place labels on Places365 (Zhou et al., 2017). The second group (M3-5) trains fENC through shot-level contrastive learning (i.e., SimCLR proposed by Chen et al. (2020a)) with different positive pair sampling strategies. Specifically, Instance (M3) takes an instance of the center shot with different augmentation, Temporal (M4) takes one randomly sampled neighbor shot as positive pair in local temporal window, and Nearest Neighbor (NN) (M5) takes the most visually similar shot among the neighbor shots as positive pair, which is also known as ShotCoL (Chen et al., 2021). The last group (M6-8) learns both fENC and fCRN through boundaryaware pretext tasks proposed in this paper. Given pre-trained representations, we train a video scene segmentation model with three different types of fCRN including MLP (Chen et al., 2021), MSLSTM (Huang et al., 2020)1 and Transformer. For fair comparison, all pre-training methods employ ResNet-50 (He et al., 2016) as the shot encoder fENC and we pre-train the models for 10 epochs.
In Table 2, we found the following observations. First, when transferring pre-trained shot representation, employing MS-LSTM and Transformer as fCRN is more effective than using MLP, as they are favorably designed to capture contextual relation between shots (see M1-6). Second, BaSSL (M7) outperforms all competing baselines (M1-5) through learning contextual representation during pre-training. Also, it turns out that transferring the representation through fCRN is important for the boundary detection task where it leads to a performance gain of 5.44%p in AP (see M6-7). Finally, learning shot-level and contextual representations is complementary to each other; that is, incorporating ShotCoL (M5) and our framework (M7) provides further improved performance (M8).

heading: 4.4 ABLATION STUDIES
text: Impact of individual pretext tasks We first investigate the contribution of individual pretext tasks. In this experiment, we train models by varying the combinations of the pretext tasks. From Table 3, we can obtain following two observations. First, when training a model with a single pretext task (P1-4), the MSM task leads to the worst performance compared to the others. This result indicates that boundary-aware pretext tasks (i.e., SSM, CGM and PP) to learn contextual relation between shots is indeed important for video scene segmentation. Second, the more pretext tasks we include during pre-training, the better the performance is, and the best performance is obtained when using all tasks (P15). This means all tasks are complementary to each other, contributing to performance gain.
1https://github.com/AnyiRao/SceneSeg/tree/master/lgss
Psuedo-boundary discovery method To check the effectiveness of DTW-based pseudo-boundary discovery, we train three models with different pseudo-boundary decision strategies—1) Random defining one randomly sampled shot in the input sequence as pseudo-boundary, 2) Fixed always taking the center shot as pseudo-boundary, and 3) Synthesized, inspired by Xu et al. (2020), synthesizing the input sequence by concatenating two sub-sequences sampled from different movies and using the last shot of the first sub-sequence as a pseudo-boundary. Table 4(a) summarizes the results. Random and Fixed pseudo-boundaries hinder the learning and degenerate the boundary detection performance. It is notable that BaSSL with Synthesized pseudo-boundaries also outperforms the pre-training baselines in Table 2, which shows the effectiveness of our framework. Finally, adopting DTW to find pseudo-boundaries achieves the best performance.
Hyperparameters We analyze the impact of two key hyperparameters: 1) the number of neighbor shotsK and 2) pre-training epochs. Table 4(b) shows that we achieve higher performance with more neighbor shots, saturating around K = 12. Table 4(c) shows the impact of longer pre-training. We find that performance increases until certain numbers (i.e., 40 epochs) and decrease afterward. We conjecture that this is partly due to overfitting to noise from incorrect pseudo-boundaries.

heading: 5 CONCLUSION
text: We present BaSSL, a novel self-supervised framework for video scene segmentation, especially designed to learn contextual relationship between shots. Through the pseudo-boundary discovery, we can define and conduct boundary-aware pretext tasks that encourage the model to learn the contextual relational representation and a capability of capturing transitional moments. Comprehensive experiments demonstrate the effectiveness of our framework and we achieve outstanding performance in the MovieNet-SSeg dataset.
Ethics statement In this paper, we introduce a novel self-supervised learning algorithm especially for video scene segmentation. The video scene segmentation technique could be applied to a wide range of applications, including preview generation for fast contents discovery, trailer generation, and minimally disruptive video-ads insertion. In addition, our proposed boundary-aware self-supervised learning algorithm can be applied to learn contextual representation of untrimmed, long videos. This would drive future research in videos to move towards leveraging the extremely large number of raw videos on the web to learn video representations. However, as a negative effect of this, it would consume large amount of computational resources. Also, as any machine learning algorithm is highly likely to be biased for training data, the learned representation would be biased and lead to an issue related to discrimination because the training data for self-supervised learning is often not validated by humans.
Reproducibility statement To make our algorithm reproducible, we present implementation details in Section 4.1 of the main paper and in Section B of the supplementary material; we also provide a pseudocode for DTW-based pseudo-boundary discovery method. In addition, we will release the code and parameters of trained models.

heading: A DATASET STATISTICS
text: Table 5 shows the data statistics of different video scene segmentation datasets. We found the limited number of datasets that provide the scene boundary annotations and, as far as we know, the MovieNet-SSeg (Huang et al., 2020) is the largest-scale video scene segmentation dataset.

heading: B ADDITIONAL IMPLEMENTATION DETAILS
text: Additional details of the shot encoder (i.e., ResNet-50) and the contextual relation network (i.e., Transformer) are as follows. For Transformer, the dimensions are set to (L = 2, H = 768, A = 8) where L, H and A mean the number of stacked transformer blocks, the dimension of hidden activation and the number of attention heads, respectively. We apply the Dropout technique (Srivastava et al., 2014) on hidden states and attention weights with a probability of 10% and use GELU (Hendrycks & Gimpel, 2016) as an activation function. For the shot encoder, each shot is given by three key-frames (i.e., Nk = 3) and a shot encoding e is computed by the averaged feature after inferring individual three key-frames by ResNet-50; note that, to speed up the training, we use randomly sample one key-frame out of three during the pre-training.
For data augmentation of key-frames in a shot, we adopt PyTorch’s torchvision package. Given a sequence of shots, we apply random crop (with resize), random flip, random color jitter and random Gaussian blur. In detail, firstly, the cropping is performed with a random size (i.e., scales between [0.14, 1.0] of the original size) and a random aspect ratio (between 3/4 to 4/3), and then the cropped one is resized to (224,224). Secondly, we apply a random horizontal flip with a probability of 50%. Thirdly, as a color augmentation, we perform a random color jitter (with a probability of 80%) and a random color dropping to gray scale (with a probability of 20%). The color strength parameters for jittering are set to {brightness, contrast, saturation, hue} = (0.2, 0.2, 0.2, 0.05). Finally, Gaussian blur is applied with a probability of 50% where a standard-deviation of spatial kernel is set to [0.1, 2.0]. Note that the same augmentations are applied to all key-frames in the input sequence Sn of shots while different color jittering is applied on individual shots. Also, for Sslown , we perform a different augmentation compared to that applied on Sn.
During the pre-training stage, the model parameters are randomly initialized and then trained using the proposed pretext tasks. We use LARS (You et al., 2017) to learn the model (except for parameters of bias and Batch-Normalization) with a mini-batch of 256 shot sequences, a base learning rate of 0.3, momentum of 0.9, weight decay of 10−6 and trust coefficient of 0.001. We pre-train the model for 10 epochs with a linear warm-up strategy for 1 epoch followed by learning rate decaying with a cosine schedule. The temperature τ in Eq. (5) is set to 0.1.
In the fine-tuning stage, we initialize the parameters of the shot encoder and the contextual relation network by that of the pre-trained ones, however, we freeze the parameters of the shot encoder following Chen et al. (2021). We fine-tune the contextual relation network and the scene boundary detection head for 20 epochs using Adam (Kingma & Ba, 2015) with a learning rate of 10−5 and a mini-batch of 1024 training examples. The learning rate is decayed with a cosine schedule without a warm-up stage.

heading: C COMPARISON WITH SHOT-LEVEL SELF-SUPERVISED LEARNING
text: As mentioned in the main paper, our approach is distinguishable from the shot-level pre-training approach (Chen et al., 2021; Qian et al., 2021) in that the objectives used in our approach (BaSSL)
is to learn contextual representations by taking neighbor shots into account. Figure 5 provides a clear summary of comparison between shot-level pre-training and our BaSSL. Firstly, shot-level pre-training takes a pair of two shots as an input while BaSSL takes a sequence of shots. Secondly, shot-level pre-training aims to train shot encoder (fENC) only, while BaSSL trains both the shot encoder and the contextual relation network (fENC and fCRN). In contrast to the shot-level pre-training that requires to train fCRN from scratch during the fine-tuning stage, BaSSL benefits from weight transfer by pre-training the parameters of fCRN with large-scale in-domain data in advance. Note that the results (M6-7) in Table 2 show that the weight transfer of fCRN is important to improve the video scene segmentation performance. Finally, the contrastive learning objective in shot-level pre-training locates the representations of two shots (query and positive) to be close to each other, whereas Shot-Scene Matching objective in our approach performs the same task but with a shot (query) and its associated scene (positive; a sequence of shots). The Table 6 summarizes the aforementioned comparisons.

heading: D ALGORITHM FOR PSEUDO-BOUNDARY DISCOVERY
text: In this section, we describe the details of pseudo-boundary discovery method applying DTW on Sn and Sslown . In practice, Sn is given as a mini-batch resulting in a tensor with a shape of (B, S, Nk, C, H , W ) where individuals mean the batch size, the number of shots in Sn (i.e., 2K + 1), the number of key-frames in a shot, channels, frame height and frame width, respectively. Then, we obtain Sslown ∈ RB×2×Nk×C×H×W that is composed of the first and last shots in Sn. We apply two different augmentation functions into key-frames in Sn and Sslown , respectively. Next, we compute encoded representation of Sn and Sslown using fENC. Note that during the pre-training stage, we randomly sample one key-frame among Nk candidates in a shot and then reshape the input tensor as (B*S, C, H , W ) or (B*2, C, H , W ) to be forwarded by the shot encoder fENC; thus the tensor shape of the encoded shot representation is given by (B, S, De) or (B, 2, De) after apply reshaping, where De means the dimension of encoded feature. Finally, given two sequences of encoded representation for Sn and Sslown , DTW provides two sub-sequences S left n and S right n and a pseudo boundary shot sn+b∗ . The algorithm 1 illustrates the details. In addition, to demonstrate the simplicity of the alignment computation using DTW, we include the PyTorch code in Listing 1. The implementation of DTW can be done in 5 lines of python code using tslearn package.
from tslearn import metrics import numpy as np def compute_dtw_path(self, seq_1, seq_2):
""" Input:
seq_1: sparse shots embedding, shape = torch.Size([2, dim]) seq_2: dense shots embedding, shape = torch.Size([N, dim]), N > 2
Output: dtw_path: output of DTW algorithm, shape = torch.Size([N, dim]) """ cost = (1-torch.bmm(seq_1, seq_2.transpose(1, 2))).numpy() dtw_path = [] for bsz in range(cost.shape[0]):
_path, _ = metrics.dtw_path_from_metric(cost[bsz], metric="precomputed") dtw_path.append(np.asarray(_path)) # torch.Size([N, dim])
return dtw_path
Listing 1: PyTorch code for alignment computation using DTW given two sequences. The tslearn package is used for DTW path calculation.
Algorithm 1 DTW-based pseudo-boundary discovery 1: Input: Shot encoder fenc, contextual relation network fCRN, and an input shot sequence Sn = {sn−K , ..., sn, ..., sn+K} centered at nth shot sn with neighbor sizeK, two image augmentation functions λ1aug, λ 2 aug.
2: (En,Eslown )← ([], []) 3: for i = n−K to n+K do 4: ei ← fENC(λ1aug(si)) // extract shot-level representations for all shots 5: En ← {En; ei} // append 6: end for 7: for i in {n−K, n+K} do 8: ei ← fENC(λ2aug(si)) // extract shot-level representations for slow sequence 9: Eslown ← {Eslown ; ei} // append
10: end for 11: Sleftn , S right n , b∗ ← DTW(En,Eslown ) // apply dynamic time warping 12: Output: Two continuous non-overlapping sub-sequences Sleftn and S right n and a pseudo boundary
shot sn+b∗ .

heading: E RESULTS ON ADDITIONAL DATASETS
text: We further compare BaSSL with shot-level pre-training baselines on additional two datasets—BBC and OVSD. Note that the train and test splits are not available and the dataset size is extremely limited (11 and 21 videos in BBC and OVSD, respectively); in addition, 2 out of 21 videos in OVSD is not available. Thus, we infer predictions using models trained on MovieNet-SSeg without finetuning on BBC and OVSD. The results are summarized in Table 7. The result shows the superiority of our method compared to shot-level pre-training baselines.

heading: F MEASURING REPRESENTATION QUALITY AT PRE-TRAINING STAGE
text: The normalized mutual information (NMI) is a metric for clustering algorithms (e.g., K-Means), which measures the clustering quality. Since clustering with good representations forms clear boundaries between different classes, NMI can be considered as a proxy to measure the quality of our pre-trained models. Specifically, we randomly sample 100 scenes from the test split of MovieNet-SSeg while we vary the length of scenes Nc ∈ {8, 16, 32}. Then, we perform K-Means clustering on Nc × 100 shot representations extracted by the pre-trained model with the number of classes K=100. We intend to form a single cluster for each scene in this formulation, assuming that high-quality representation for movie scene segmentation would locate the shot features within the same scene close to each other. Considering the randomness in the K-Means clustering and scene sampling, we report the averaged score from five trials.
In Table 8, we compare the NMI score between different pre-trained models; SimCLR (NN) is our SimCLR version implementation of ShotCoL. The result shows that BaSSL outperforms the shot-level pre-training baselines and the model pre-trained using ImageNet dataset. With respect to different scene lengths (Nc; the number of shots included in a single scene), we found our BaSSL is more robust than the other baselines. Since the visual diversity across the shots increases as the scenes become longer (Nc=8→ 32), it is natural that the NMI score for each baseline is degraded. However, it is remarkable that, by increasing the number of shots from 8 to 32, the performance of BaSSL drops only -0.68% while the other baselines suffer from severe degradation. This demonstrates the effectiveness of BaSSL in maximizing intra-scene similarity.
In addition, we perform ablation study of our algorithm by adding pretext tasks one by one, and measure the corresponding NMI scores. The result in Table 9 shows that better NMI score is achieved as more pretext tasks are combined together. This tendency is also observed in our ablation in Table 3, which indicates the NMI score of pre-trained models is highly correlated with the final performance after the fine-tuning.

heading: G QUALITATIVE ANALYSIS
text: Visualization of similarities between consecutive shots To qualitatively check the effect of individual pretext tasks, we visualize the matrix of cosine similarity between shot representations from the randomly sampled 16 consecutive shots in Figure 6. The shot representations are computed by models without the fine-tuning in order to solely focus on the behavior of each objective at the pretraining stage. When the MSM is used only, approximately three clusterings are shown, but similarity around boundaries is smoothed. Next, when we add PP, dissimilarities around the boundaries are to be sharpened. Then, with additional CGM, the clusters are more clearly obtained. Finally, adding SSM makes the similarity of shots within the same cluster higher (i.e., more yellow ones).
Pseudo-boundaries We compare the quality of discovered pseudo-boundaries with the ground truth scene boundaries in Figure 7. In most cases, we observe the pseudo-boundaries identified by
the DTW algorithm are successfully located in close distance with the ground truth ones. This result validates our idea considering the problem of discovering pseudo-boundary as a temporal alignment problem between two sequences with different frequencies (Sn and Sslown ). At the same time, we illustrate the failure cases. Although discovered pseudo-boundary does not match the ground truth in this case, we figure the determined boundary is not always arbitrary. For example, the mismatch is often caused by the noise existing in the ground truth (see the first row in the failure cases). On the other hand, in case all shots are visually similar (see the third row in the failure cases), the DTW solely relying on the visual modality fails to find the correct boundary.
Predicted scene boundaries The figure 8 illustrates the scene boundary predictions of different models. Comparing with the baselines, we observe that our approach, BaSSL, shows qualitatively better performance for video scene segmentation. On the other hand, we observe the oversegmentation issue in many cases using any compared methods (including ours). Our finding implies that achieving the highest recall only does not guarantee the highest performance in practice. We reckon that further studies on this over-segmentation problem would be a highly important topic when it comes to real-world application.

Review summaries:
The paper introduces a boundary-aware self-supervised learning framework for video scene segmentation. Based on the contrastive learning protocol, the authors employ DTW to generate dimidiate video segmentation from unlabeled video. Using these segments, the authors employ four loss functions to pre-train a backbone network.

This paper introduces a self-supervised learning method for the video scene segmentation task, which needs to temporally localize scene boundaries in a video. Their proposed method defines pseudo-boundary from a sequence of shots by splitting it into two sub-sequences and train a model to predict the pseudo-boundary with three novel tasks: Shot-Scene Matching (SSM), Contextual Group Matching (CGM), and Pseudo-boundary Prediction (PP). Experiments on pre-training and transfer learning show that this method is critical to improving the video scene segmentation performance, and it achieves the new state-of-the-art on the MovieNet-SSeg benchmark.

This paper presents a boundary-aware self-supervised learning framework for the task of video scene grouping.  During the pre-train stage, the authors first apply DTW to find a pseudo boundary in a given video sequence, and then applies Shot-Scene Matching and Contextual Group Matching to optimize the intra-scene and inter-scene distribution of features. Moreover, Pseudo-boundary Prediction and Masked Shot Matching are utilized to help the model identify transitional moments.