Boundary-aware Pre-training for Video Scene Segmentation
Authors: Jonghwan Mun
Minchul Shin
Gunsoo Han
Sangho Lee
Seongsu Ha
Joonseok Lee
Eun-Sol Kim
Venue: ICLR 2022 Submitted (2022)
Keywords: Temporal Segmentation
Video Scene segmentation
Self-supervised Learning

Abstract: Self-supervised learning has drawn attention through its effectiveness in learning in-domain representations with no ground-truth annotations; in particular, it is shown that properly designed pretext tasks (e.g., contrastive prediction task) bring significant performance gains for a downstream task (e.g., classification task). Inspired from this, we tackle video scene segmentation, which is a task of temporally localizing scene boundaries in a video, with a self-supervised learning framework where we mainly focus on designing effective pretext tasks. In our framework, we discover a pseudo-boundary from a sequence of shots by splitting it into two continuous, non-overlapping sub-sequences and leverage the pseudo-boundary to facilitate the pre-training. Based on this, we introduce three novel boundary-aware pretext tasks: 1) Shot-Scene Matching (SSM), 2) Contextual Group Matching (CGM) and 3) Pseudo-boundary Prediction (PP); SSM and CGM guide the model to maximize intra-scene similarity and inter-scene discrimination while PP encourages the model to identify transitional moments. Through comprehensive analysis, we empirically show that pre-training and transferring contextual representation are both critical to improving the video scene segmentation performance. Lastly, we achieve the new state-of-the-art on the MovieNet-SSeg benchmark. The code will be released.

Under review as a conference paper at ICLR 2022
BOUNDARY-AWARE SELF-SUPERVISED LEARNING FOR VIDEO SCENE SEGMENTATION
Anonymous authors Paper under double-blind review
ABSTRACT
Self-supervised learning has drawn attention through its effectiveness in learn- ing in-domain representations with no ground-truth annotations: in particular, is shown that properly designed pretext tasks (e.g., contrastive prediction task) bring significant performance gains for a downstream task (e-g., classification task). In- spired from this, tackle video scene segmentation, which is task of temporally localizing scene boundaries a video, with a self-supervised learning framework where mainly focus designing effective pretext tasks. In our framework, we discover a pseudo-boundary from a sequence of shots by splitting it into two continuous, oon-olella sub-sequences and leverage the bseudo-boundee facilitate the pre-training. Based on this, we introduce three novel boundary-aware pretext tasks: 1) Shot-Scene Matching (SSM). 2) Contextual Group Matching (CGM) and 3) Pseudo-boundary Prediction (PP): SSM and CGM guide the model maximize intra-scene similarity and inter-scene discrimination while PP en- courages the model to identify transitional moments. Through comprehensive analysis, we empirically show that pre-training and transferring contextual repre- sentation are both critical to improving the video scene segmentation performance. Lastly, we achieve the new state-of-the-art on the MovieNet-SSeg benchmark. The code will be released.
INTRODUCTION
The video scene segmentation is a task of identifying scene boundaries from video where scene defined as semantic unit for making a story and composed of series of semantically cohesive shots-a a set of frames captured by the same camera during uninterrupted period of time the same context. Localizing scene boundaries is a significant step towards the high-level video understanding because dividing a long video into set of meaningful scenes enables models to inspect the individual incidents from complex story.
One of the biggest challenges with temporal semantic segmentation is that it is not achieved simply by detecting changes visual cues. As shown in Figure [[a), we present an example of nine shots, all of which belong to the same scene, where that the overall visual cues within the scene do not stay the same but rather change repeatedly when each character appears. On the other hand, as presented in Figure [[b), the other example shows two different scenes which contain visually similar shots (highlighted in blue) where the same character appears in the same place. Therefore, is expected that two adjacent scenes which share shots with similar visual cues need to be contextually discriminated. From this observation, it is important for the video scene segmentation task to model contextual relation between shots by maximizing 1) intra-scene similarity (i.e., the shots belonging to the same scene should be close to each other), and 2) inter-scene discrimination across two adjacent scenes (i.e., two neighbor shots across the scene boundary should be distinguishable).
characters are talking
the phone. We can
Supervised learning approaches (e.g., Rao et al. (2020) are clearly limited due to the lack of large- scale datasets with reliable ground-truth annotations. Recently, self-supervision (Chen et al. 2220aa
Caron et al.) 2020 He et al. 2020 Roh et al. 2021) spotlighted through its effectiveness in
learning in-domain representation without relying on costly ground truth annotations. The self-
supervised learning methods (Chen et al. 2021 Feichtenhofer et al. 2021 Dave et al.) 2021] Qian et al. 2021) in the video domain are often designed to learn spatio-temporal patterns in short clips

---

Under review as a conference paper at ICLR 2022
Dissimilar visual
Similar visual different scenes
Boundary
Examples of the video scene segmentation. In each row, we visualize the shots including similar visual cues (e.g., characters, places, etc.) with the same colored border.
Figure 1:
(e.g., shots in movies). This kind of learned representation is generic and can be applied to many video understanding tasks (e.g., action classification). However, such representation is not sufficient for video scene segmentation because this task requires not only good representation for individual shots but also contextual representation considering neighboring shots at a higher level as illustrated in Figure [ Motivated by this, we set our main goal to design effective self- seff-supervised objectives (ie.. pretext tasks) that maximize intra-scene similarity as well as discriminate shots from different scenes. For the purpose, this raises penetrating question: how can we design boundary- pretext tasks without access to ground truth boundary annotations?
We introduce novel Boundary-awe aware Self-Superviser Learning (BaSSL) framework. Our main idea of BaSSL to localize a pseudo-boundary, which is obtained by dividing the input sequence of shots into two semantically disjoint sub- uub-sequenees, and use it to define pretext tasks that beneficial to the video scene segmentation task. On top of the discovered two sub-sequences and a beeudo-boundary, three boundary-awan pretext tasks are proposed: 1) Shot-Scene Matching (SSM); 2) Contextual Group Matching (CGM); and 3) Pseudo- -ouundrry Prediction (PP). Note that SSM and CGM encourage the model to maximize intra-scene similarity and inter-scene discrimination while PP enables the model to learn the capability of identifying transitional moments. In addition, perform Masked Shot Modeling (MSM) task inspired by (unn et al. 2019a) further learn temporal relationship between shots. The comprehensive analysis demonstrates the effectiveness of the proposed framework (i.e., pre-training of contextual relationship between shots) as well as the contribution of the proposed individual components (i.e., the algorithm for pseudo-boundary discovery and boundary-aware pretext tasks).
Our main contributions are summarized as follows: (i) We introduce a novel boundary- pre-training framework which adopts dynamic time warping (DTW) algorithm to identify pseudo- boundaries and use them as self-supervision to facilitate the pre-training: (rd) we propose three cunndar--aware pretext tasks, which are carefully designed to learn essential capabilities required for the video scene segmentation task; (iit) we perform extensive ablations to demonstrate the effec- tiveness of the proposed framework, including the observation that our framework is complementary to the existing framework; (it) we achieve the new state-of-the-ard the MovieNet-SSS bench- mark while outperforming existing self-supervised learning- baasdd methods by large margins.
2 RELATED WORK
Video scene segmentation approaches formulate the task as a problem of temporal grouping of shots. In this formulation, the optimal grouping can be achieved by clustering- (Rui et al.
1998 Rasheed & Shah 2003 2005) Chasanis et al. 2008), dynamic programming-basse (Han & Wu 2011 Tapaswi et al. 2014 Rotman 2017) or multi- input-based (Liang al. 2009
Sidiropoulos et al.] 2011] methods. However, the aforementioned methods have been trained and evaluated on small-scale datasets such as OVSD (Rotman et (016) and BBC (Baraldi et al. 2015) which can produce a poorly generalized model. Recently, Huang et al. (2020) introduce a large- scale video scene segmentation dataset (i.e., MovicNet-SSeg) that contains hundreds of movies. Training with large-scale data, |Rao et al. (2020) proposes strong supervised baseline model that performs a shot-level binary classification followed by grouping using the prediction scores. In addition, Chen et al. (2021) proposes shot contrastive pre-training method that learns shot-level representation. We found ShotCoL (Chen et al. (02T) to be the most similar work to our method. However, method is different from ShotCol. that we specifically focus learning contextual
2

---

Under review as a conference paper at ICLR 2022
representations by considering the relationship between shots. We refer interested readers to the supplementary material for a more detailed amalysis of this.
Action segmentation in videos is of the related works for video scene segmentation, which identifies action labels of individual frames, thus divides a video into a series of action segments. Su- pervised methods (Lea al. 2016 [arha & Gall 2019) proposed CNN-based architectures to effec- tively capture temporal relationship between frames in order to address an over-segmentation issue. As frame-level annotations are prohibitively costly acquire, weakly supervised methods (Chang
2019; Li et al., 2019 Li & Todorovic| 2020; [Souri et al. [2021; Shen et al., [021]] Zhukov et al.
al.
2019 Fried et al., 2020) have been suggested to use an ordered list of actions occurring in
video as supervision. Most of the methods are trained to find (temporal) semantic alignment
between frames and a given action list using an 1IMM-based architecture (Kuehne et al.
2018),
dynamic programming-baset assignment algorithm (Fried et al.) 2020) or a DTW-based tempo- ral alignment method (Chang et al. 2019). Recently, unsupervised methods (Kumar et al. 2021
Wang et al.] 2021, Kukleva et al. 2019 Li & Todorovic 2021 VidalMata et al., 2021) have been further proposed; in a nutshell, clustering based prototypes are discovered from unlabeled videos, then the methods segment the videos by assigning prototypes (corresponding to one of the actions) into frames. In contrast to the action segmentation task that is limited to localizing segments each of which represents single action within an activity, video scene segmentation requires localizing more complex segments each of which may be composed of more than two actions (or activities).
Self-supervised learning in videos has been actively studied for the recent years with approaches proposing various pretext tasks such as future frame prediction (Srivastava et al. 2015 Vondrick
ctt al. 2016 AAhan et al. 2018). temporal ordering of frames (Misra et al. 2016 Lee et al.
2017
et al., 2019), geometric transformations prediction (Jing & Tian 2018) colorization of videos (Von-
drick et al.
2018) and contrastive prediction (Feichtenhofer et al.) 2021; Qian et al.)
2021; Dave
al. 2202 In addition, CBT (Sun et al.) 2019a b) proposes a pretext task of masked frame model- ing to learn temporal dependency between frames (or clips). Note that since most of those methods proposed for the classification task, they would be sub-optimal to the video scene segmentation task. On the other hand, BSP (Xu et al. (020) proposes boundary-sene sensitive pre-text tasks based on synthesized greudo-boundarde that are obtained by concatenating two clips sampled from dif- ferent videos. However, strictly speaking, BSP is not a self-supervised learning algorithm since it requires video-level class labels to synthesize pseudo-boundaries; the proposed pretext tasks not applicable to videos such as movies that are hard to define semantic labels. Also, note that we em- pirically show that pseudo-boundaries identified by our method are more effective for pre-training than synthesized sreudo-boundaries.
3 BOUNDARY-AWARD SELF-SUPERVIES LEARNING (BASSL)
In this section, we introduce our proposed approach, Boundary-aware Self-Supervised Learning (BaSSL). We start with the problem formulation followed by the model overview. Then, we describe our novel coundary-aware pretext tasks for pre- mre-trannnn..
3.1 PROBLEM FORMULATION
Terminologies A video (e.g., documentaries, TV episodes and movies) is a sequence of scenes, defined as a semantic unit for making a story, A scene is a series of shots, which is a set of frames physically captured by the same camera during an uninterrupted period of time.
Problem Definition Given a video, which contains series of N shots (Si....,SN) with class labels (y,,,.., yN where y E (0. 1} indicating if it is at the scene boundary (more precisely, if is the last shot of a scene), the video scene segmentation task is formulated as a simple binary classi- fication problem at individual shot level. By definition, a scene boundary is where the semantic of a shot is considerably different from its (one-way) neighbors. Thus, it is in nature important to capture and leverage contextual transition across the scenes. Consequently, it is a common practice that the information of the neighbor shots leveraged together when determining scene boundaries. With this formulation, existing supervised learning approaches typically train a parameterized (0) model by maximizing the expected log- lielliiel
3

---

Under review as a conference paper at ICLR 2022
Dynamic Time Warping
SSM CGM PP MSM
Boundary (1)? not (0)?
Pse Porudo-boundan boundrry Discovery
CRN (sss
CRN (Su)
OOOOOOOOO
transfer
O OO0O
Shot Encoder Uexc)
Shot Encoder Usscl
000000000
0000 0000
Pseudo boundory Shot IM
Pre-training Stage
Fine-tuning Stage
Figure 2: Overall pipeline of our proposed framework, BaSSL.
negative pair
non nom-boundary (0)
boundary (1)
pull
CRN (/w)
supppocling
CRN
CRN [/aw]
masking
Shot Encoder (Fve)
Shot Encoder (fee)
Shot Encoder (fox)
Shot Encoder (fox)
Shot Scene Matching (SSM)
Contextual Group Matching (CGM)
Pseudo-boundary Prediction (PP)
Masked Shot Modeling (MSM)
Figure 3: Illustration of four pre trainrain pretext tasks.
0" = arg max E [log pe (vn
(I)
where S., = (Sn Sn Sn+ K is a set of 2K + shots centered at n shot Sn, and K is the number of neighbor shots before and after S.- Note that each shot is given by set of N₂. key- frames, resulting in a tensor with size of (N₂, C, H, W) where C, and W are the RGB channels, the frame height and the frame width, respectively.
3.2 MODEL OVERVIEW
Our method based on two-stage training following common practice (Chen et al. 202I): pre- training on large-scale unlabeled data with self-supervision and fine-tuning relatively small la- beled data via transfer learning. Our main focus is in the pre-training stage, aiming at designing effective pretext tasks for video scene segmentation.
As illustrated in Figure our model consists of two main components: 1) shot encoder embedding shot by capturing its spatio-temporal patterns, and 2) contextual relation network (CRN) capturing relationship between shots. Given sequence S,, = {sn- .n., Sut K) of shots centered at Sn, two-level representations are extracted as follows:
en fEN and C, = FCRN(EE,)))
where SExc: IRNxxC R. and SCRN: R(2K+1)xDr - R(2K+1)xD= represent the shot encoder and the contextual relation network while De and D. mean dimensions of encoded and contextualized features, respectively. en an encoding of shot S, by fENc while E,, {en-K, .... e...... en+k] }o  and C, = {Cn-K,..., Cas o., Cn+K) correspond to the input and output feature sequence for fCRN, respectively.
On top of encoded shot representations, BaSSL extracts pseudo-boundee (left box of Figure 2 to self-supervise the model instead of relying on ground-trun -ruth annotations. To be specific, we leverage the dynamic time warping technique to divide the input sequence of shots into two semantically disjoint sub-sequences and output pseudo- bounday. (See Section 3.3 for more details.)
Then, as presented in Figure 3 using the discovered pseudo-boundary, we devise three novel boundary-aware pretext tasks in Section 3.4 1) Shot-Scene Matching to match shots with their associated scenes, 2) Contextual Group Matching to align shots whether they belong to the same scene or not and 3) bouuddryyundry Prediction to capture semantic changes. addition, we adopt the masked shot modeling in CBT (Sun et al.) 2019a) to further learn temporal relationship between shots. After trained with the four pretext tasks, the model tuned with labeled video scene segmentation data, (See Section 3.5 for more details.)
4

---

Under review as a conference paper at ICLR 2022
(a)
(b)
Pocudo boundary
Psoudo Scone
Pscudt Soone B
Figure 4: An example in each row shows sequence of shots sampled from the same scene where there exists no ground-truth scene-level boundary. Our method finds houuda-yyundar shot (high- lighted in red) that divides sequence into two pseudo-scenes (represented by green and orange bars, respectively) so that semantics (e.g., places, characters) maximally changes.
3.3 SAUUDO-BOUNDAYY DISCOVERY
During the pre-training, no scene boundary annotation is available for the input sequence S, of shots. Thus, infer gecudobboundaries which serve they are ground truth labels for boundary- relevant pretext tasks. In addition, we simplify the problem to always extract single boundary given an input sequence. This is reasonable because even a sequence of shots without strong scene- level semantic transition, there always exists shot within the sequence across which the semantic transition maximum, and we use this shot as pseudo-boundary, To support our claim, we show examples in Figure where we intentionally infer pseudo-boundary based on a sequence of shots sampled from the same scene (that is, no scene boundary exists between shots according to the ground truth); we observe that the resulting two sub-sequences are cognitively distinguishable.
This simplified problem, dividing the input sequence S,, into two continuous, non-overlapping sub- sequences Sleft and Sright can be seen as a temporal alignment problem between S,, and Sslow; specifically, observing the first shot should belong to Sieft and the last one to S define Sslow {Sn- K, Sn+ K J. which can be seen as a same video with S,, with lower sampling frequency. Finally, the task becomes aligning intermediate shots either to Steft or sright preserving continuity.
Under the problem setting, we adopt dynamic time warping (DTW) (Berndt & Clifford 1994) to find the optimal temporal alignment between S, and Syllow In detail, DTW solves the following optimization problem using dynamic programming to maximize semantic coherence the resulting two sub- sequences among all possible boundary candidates:
b
arg max
sim(c, K. Cn+i)
sim(Ca K2 Ca4K)Cn4j)
b+
K
j=b+1
(3)
where b is the candidate boundary offset, b" is the optimal boundary offset, and sim y) computes cosine similarity between encodings of the given two shots. Two sub-sequences are inferred as Sleft = (Sn=KK....S2+bb
x
|x||||>|
and Stight {Sn46* Sn-kk Sn+h- is the pseudo- boundary shot, which is the last shot of Slen More examples of pseudo-boundaries identified by our algorithm is presented in Figure 7 of the supplementary material. The results are used for learning boundary-awe aware pretext tasks, which will be described below.
3.4 PRE-TRAINING OBJECTIVES
As illustrated in Figure 3 we adopt four pretext tasks for pre-training-1) shot-scene matching, 2) contextual group matching, 3) pseudo-boundary prediction, and masked shot modeling- and the final pre-training loss is defined by
Cpretrain LSsM + LOOM + Lpp LMSM-
(4)
Shot-Scene Matching (SSM) The objective of this task is make the representations of shot and its associated scene similar to each other, while the representations of the shot and other scenes dissimilar. In other words, SSM encourages the model to maximize intra-scene similarity, while
5

---

Under review as a conference paper at ICLR 2022
minimizing inter-scene similarity. Considering the splitted two sub- sequences (S), and S. as pseudo-scenes, we train the model using the InfoNCE loss (Oord et al.) 2018):
LSSM LxCE (hssm(en- (rsM + LNCE (hssM (Cn+ K), hIss (r(S)""))) (5)
exp(sim(e, r)/7)
(xcr(e,r) log
exp(sim(e.r))/)) + - exp(sim(e, r)/T) + =E exp(sim(e, F)/T)
(6)
where hSsM is a SSM head of a linear layer, T is a temperature hyperparameter and r(S) means scene-level representation; we use the averaged encoding of shots in the sub-sequence S. Nº and N,, Eq. are constructed using other shots and pseudo-scenes a mini-batch, respectively.
Contextual Group Matching (CGM) Since directly matching representations of shots and scenes would not be effective when the scenes are composed of visually dissimilar shots, CGM is intro- duced to bridge this gap. Similar to SSM, CGM is also designed maximize intra-scene similarity and inter-scene discrimination. However, CGM measures semantic coherence of the shots rather than comparing visual cues. With CGM, the model learns to decide if the given two shots belong to the same group (i.e., scene) or not. In detail, we use the center shot Sn in the input sequence as the anchor and construct a triplet of (Sn. Spos, Sneg). We sample each shot from sleft and S the one sampled within the same sub-sequence with s, is used as the positive shot Sposs while the other as the negative Speg. The CGM loss is defined using a binary cross-entropy loss as follows:
(7) where lCGM is a CGM head taking two shots as input and predicting a matching score. Ca. Cpos and Gneg the contextualized features by JCRN for the center, positive and negative shots, respectively.
LCGM
log (hCaM (Co, Cpos) log (1 fccoM( Cuog)).
Pseudo-boundary Prediction (PP) Through the above two pretext tasks, our model learns the contextual relationship between shots. In addition these, design extra pretext task, PP, which is more directly related to boundary detection; PP makes the model have a capability of identifying transitional moments that semantic changes. Based on the aseudo-boundary shot and one randomly sampled non-boundary shot, the PP loss is defined as a binary cross-entropy loss:
Lpp log (hpp(Cn++b)) )) log (1 hupp(C)))
(8)
where hpp is a PP head that projects the contextualized shot representation to a probability distri- bution over binary class. Ca+a- and с₁ indicate the contextualized representation from fCRN for the pseudo-boundary shot Sn46* and randomly sampled non-boundary shot Sg, respectively.
Masked Shot Modeling (MSM) Inspired by masked frame modeling (Sun et al. 2019a b), adopt the MSM task whose goal is to reconstruct the representation of masked shots based on the their surrounding shots. In this task, given a set of encoded shot representations, we randomly apply masking each of them with probability of 15%. For a set M of masked shot offsets, we learn to regress the output on each masked shot to its encoded shot representation, which is given by
LMSM = Σ Icm. hMMM(
MEM
where MMMM is a MSM head to match the dimension of contextualized shot representation with that of encoded one. em and C. denote the encoded representation by fENC and contextualized representation by fCRN for a masked shot Sm, respectively.
3.5 FINE-TUNING FOR SCENE BOUNDARY DETECTION
Recall that formulate the video scene segmentation a binary classification task to identify con- textual transition across the scene. Different from the pre-training stage, given an input sequence of shots S,, we employ scene boundary detection head hSED to infer a prediction from the con- textualized representation (c,,) for the center shot sn, Following Chen et al.] (2021), we freeze the parameters of the shot encoder and then train only the contextual relation network and the scene boundary detection head using a binary cross-entropy loss with the ground truth label yn as follows:
Limetune Un log log(hsBo (c,))( + (1 yn)) log(1 hc,))),)))
(10)
Note that individual shots are decided to be a scene boundary when its prediction score is higher than pre-defined threshold (set to 0.5).

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison with other algorithms. i and denote that the numbers are copied from (Rao et al. 2020) and (Huang et al.] 2020), respectively. indicates the methods exploiting additional modalities or semanties (e.g., audio, place, cast). The best numbers are highlighted in bold.
Method
AP (t)
mloU (t)
AUC-ROC (t)
FI (t)
Supervised Learning
Siamese (Baraldi al. 20155]
35,80
39.60
MS-LSTM (Huang et al. 2020
46.50
46.20
LGSS (Rao al., 20201
47.10
48.80
Unsupervised Learning
GraphCut ut (Rasheed & Shah 2005
14.10
29.70
SCSA (Chasanis et al. 2008
14.70
30.50
DP (Han & WT 2011
15,50
32.00
Story Graph Tapuswi et al. 2014)
25.10
35.70
Grouping Rolman et at, 2017]
33.60
37.20
BaSSL w/o line-tuning (10 epuchs)
31.55
39.36
71.67
32.55
Self-supervised Learning
ShotCol. (Chen et al. 2021
53.40
BaSSL epochs)
56.26 10.04
49.50
10.11 50.69
90.27 10.02
45.70 10.24
BaSSL (40 epochs)
57.40 10.08
10.45
90.54 10.03
47.02 -0.87
4 EXPERIMENT
4.1 EXPERIMENTAL SETTINGS
Dataset We evaluate our proposed method on the MovieNet-SSeg dataset (Huang et al., 2020) that is sub-dataset of MovieNet, containing 1,100 movies with 1.6M shots. Note that only 318 out of 1,100 movies have scene boundary annotations, which are divided into 190, 64, and 64 movies for training, validation, and test split, respectively. Following Chen al. (202T), we the entire 1, 1,100 movies with no ground truth labels for the pre-training and fine-tune the model on the training split. The performance measured on the test split.
Metric Following Huang al. (2020), compare algorithms using Average Precision (AP) and
mloU that measures the averaged intersection over union (IoU) between predicted scene segments and their closest ground truth scene segments. Also, we adopt Fl score and AUC-ROC as additional
evaluation metrics. Note that contrary to the previous works (Rao et al. 2020; Chen et al. 202[) that
report recall, we use F1 score consider for balanced comparison between precision
recall. In
addition, we report Meta-Sum metric inspired by the works (Chen et al.) 2020b Li et al. 202 for easy and straightforward comparison of algorithms.
Implementation details We employ ResNet-50 (He et al., 2016) and Transformer (Vaswani al., 2017) networks the shot encoder and the contextual relation network, respectively. For both pre-training and fine-tuning stages, we cross-validate the number of neighbor shots among K = (4, 8, 12, 16) and K is selected due its good performance and computational efficiency. In all experiments, given a pre- pre-trdined model, we fine-tune the model 5 times with different random seeds and report their average score and standard deviation. Note that more details are presented in supplementary material.
4.2 COMPARISON WITH SFATE-OF-THE-AAE METHODS
We compare our method, BaSSL, with 1) supervised methods including Siamese (Baraldi et al.) 2015), MS-LSTMM (Huang et al. 2020) and LGSS (Rao et al., 2020). 2) unsupervised methods in- cluding GraphCut (Rasheed & Shah 2005). SCSA (Chasanis et al. 2008). DP (Han & Wu 2011) StoryGraph (Taaaswi et al. 2014) and Grouping (Rotman et al. 2017). and 3) self-supervised meth- the downstream task, BaSSL
ods including ShotCoL (Chen et al. 2021). Without fine-tuning can be seen as an unsupervised model in that it is trained to predict the pseudo-boundary by the PP task. Table [ summarizes comparison against state-of- oftte----hh--rr the-art methods. BaSSL without fine-tuning shows competitive or outperforming performance based only on basic visual features compared to competing unsupervised methods. Furthermore, fine-tuning BaSSL with ground-truth scene bound-
7

---

Under review as a conference paper at ICLR 2022
Table 2: Average precision (AP) comparison with pre-training baselines. Note that SimCLR (NN)
corresponds to our reproduced ShotCoL using SimCLR as the constrastive learning scheme.
Pre-training
Transfer
Architecture of feRY during fine-tuning
Method
feNC. fCRN
fENC
JeHN
MLP
MS-LSTM
Transformer
Supervised pre-training using image dataset
M1
ImageNet
43.12 +0.14
45.10
+0.55
47.13
+1 1.004
M2
Places365
43.82 +0.10
45.87
+0.40
48.71
+0.50
Shot-level pre-training
M3
SimCLR (instance)
45.60 +0007
49.09
=0.24
51.51
10.31
M4
SimCLR (temporal)
45.55 10.11
49.24
10.26
50.05
10.78
MS
SimCLR (NN)
45.99 10.13
50.73
10.19
51.17
10.69
Boundary-aware pre-training
M6
BaSSL
46.53 +0.11
50.58 +0.14
50.82
+0.69
M7
BaSSL
56.26
0.04
M8 M5+M7
56.86 =0.01
aries, AP is improved by 24.71%p and BaSSL outperforms all other algorithms. Finally, through longer pre-training (40 epochs), BaSSL surpasses the state-of-the-ar method (i.e., ShotCoL) by a large margin (4.00%p in AP).
4.3 COMPARISON WITH PRE-TRAINING BASELINES
We perform extensive experiments to compare BaSSL with the other pre-training baselines that learn shot-level representation by fENC- In the experiments, we compare the following three types of pre- training approaches: The first group (M1-2) trains fENC using image-level supervision with object labels on ImageNet (Deng et al.) 2009) or place labels on Places365 (Zhou et al. 2017). The second group (M3-5) trains jENC through shot-level contrastive learning (i.e.. SimCLR proposed by Chen al. (020a)) with different positive pair sampling strategies. Specifically, Instance (M3) takes an instance of the center shot with different augmentation, Temporal (M4) takes one randomly sampled neighbor shot as positive pair in local temporal window, and Nearest Neighbor (NN) (M5) takes the most visually similar shot among the neighbor shots as positive pair, which is also known as ShotCoL (Chen et al.) (021). The last group (M6-8) learns both fENC and fCRN through boundary- aware pretext tasks proposed in this paper. Given pre-trained representations, we train video scene segmentation model with three different types of fCRN including MI. (Chen et al., 2021), MS- 2020 and Transformer. For fair comparison, all pre-training methods employ 2016) as the shot encoder feNc and we pre-train the models for 10 epochs.
LSTM (Huang al.
ResNet-50 (He et al.
In Table 2 we found the following observations. First, when transferring pre-trained shot represen- tation, employing MS-LSTM and Transformer as JCRN is more effective than using MLP, as they are favorably designed capture contextual relation between shots (see M1-6). Second, BaSSL (M7) outperforms all competing baselines (M1-5) through learning contextual representation during pre-training. Also, it turns out that transferring the representation through fCRN important for the boundary detection task where it leads to a performance gain of 5.44%p in AP (see M6-7). Finally, learning shot-level and contextual representations complementary to each other; that is, incorpo- rating ShotCoL (M5) and our framework (M7) provides further improved performance (M8).
4.4 ABLATION STUDIES
Impact of individual pretext tasks We first investigate the contribution of individual pretext tasks. In this experiment, train models by varying the combinations of the pretext tasks. From Table we can obtain following two observations. First, when training a model with a single pre- text task (P1-4), the MSM task leads the worst performance compared the others. This result indicates that boundary-aware pretext tasks (i.e., SSM, CGM and PP) to learn contextual relation between shots is indeed important for video scene segmentation. Second, the more pretext tasks we include during tre-training. the better the performance is, and the best performance is obtained when using all tasks (P15). This means all tasks are complementary to each other, contributing to performance gain.
8

---

Under review as a conference paper at ICLR 2022
Table 3: Ablation study varying combinations of pretext tasks for pre-training. The best scores are highlighted in bold.
Pretext Tasks
Evaluation Metric
SSM CGM PP MSM
AP
mloU
AUC-ROC
Fl
Sum
Pl
42.37 10.29
40.12
+0.50
84.11 10.15
30.83
=0.79
197.63
P2
36.76 +0.02
40.59
+0.18
82.06
+0.04
30.94
=0.32
190.35
P3
36.55 +0.04
39.58
+0.05
81,36 +0.03
29.96
-00.44
187.45
13.33 10.23
29.80 10.39
64.65 10.98
=0.39
126.45
P5
55.77 10.05
48.19 10.21
90.19 10.03
43.17 =0.39
237.32
P6
56.04 10.08
49.00 10.16
90.13 10.02
10.29
239.91
P7
38.09 +0.03
41.25
82.85 #0.01
32.24 =0.24
195.43
PS
54.39 +0.07
$7.5440.111
89.72 +0.03
42.48 =0.22
234.13
P9
39.49 +0.04
41.71 +0.12
83.27 +0.02
32.85 -0.20
197.32
PIO
38.53 10.07
40.85 10.15
82.78 10.04
31.47 00.16
193.63
PII
41.02 10.07
10.10
83.79
31.53
197.23
56.10 10.08
49.10 40.17
90.09 10.03
45.42 =0330
240.71
P13
56.20 +0.06
48.00 +0.17
90.13 +0.01
43.24 00277
237.57
P14
4
56.26 10.02
48.42 40.33
90.25 10.01
43.98
00.58
238.91
P15
56.26 #0.04
49,50 +0.11
90.27 #0.02
45.70 0.24
241.73
Table 4: Ablations to check the impact of pseudo- discovery strategies, the number of neighboring shots and longer pre-training. The best scores are in bold.
Eseudo-boundaal
AP
Neighbors
AP
Epochs
AP
Random
46.64 +0.37
4
55.98 =0. 10
10
56.26 =0.04
Fixed
49.53 +0.32
56.26 10.04
20
56.74 +10.44
Synthesized
54.61 +:0.33
56.29 10.03
56.74 +0.07
DTW (ours)
56.26 10.4
16
55.31 #0.04
40
57.40 10.08
50
57.15 10.08
(a) Performance comparison depend- discovery
(b) Performance comparison when varying the number of neighbor shots.
ing on the methods.
(c) Performance comparison with respect to the number of pre-training epochs.
Psuedo-boundary discovery method To check the effectiveness of DTW-based pseudo-boundar discovery, we train three models with different defining randomly sampled shot in the input sequence pseudo-boundary, 2) Fixed always taking the center shot as pseudo-boundary. and 3) Synthesized,, inspired by Xu et al. (2020), syn- thesizing the input sequence by concatenating two sub-sequence, sampled from different movies and using the last shot of the first sub-sequence as a houudd--bundary... Table 4[a) summarizes the results. Random and Fixed beuudi-buudarris hinder the learning and degenerate the boundary de- tection performance. It is notable that BaSSL. with Synthesized secudobboundaries also outperforms the pre-training baselines in Table [ which shows the effectiveness of our framework. Finally, adopting DTW to find pseudo- bounddrss achieves the best performance.
decision strategics-1) Random
Hyperparameters We analyze the impact of two key hyperparameters: 1) number of neighbor shots K and 2) pre-training epochs. 4 shows that we achieve higher performance with more neighbor shots, saturating around K 12. Table 4(c) shows the impact of longer pre-training. We find that performance increases until certain numbers (i.e., 40 epochs) and decrease afterward. We conjecture that this is partly due to overfitting to noise from incorrect pscudo-boundarics.
CONCLUSION
We present BaSSL, a novel self-supervised framework for video scene segmentation, especially de- signed to learn contextual relationship between shots. Through the pseudo-boundans discovery, can define and conduct boundary-aware pretext tasks that encourage the model to learn the contex- tual relational representation and capability of capturing transitional moments. Comprehensive experiments demonstrate the effectiveness our framework and we achieve outstanding perfor- mance in the MovieNet-SSeg dataset.

---

Under review as a conference paper at ICLR 2022
Ethics statement In this paper, introduce novel self-supervised learning algorithm espe- cially for video scene segmentation. The video scene segmentation technique could be applied to wide range of upplications, including preview generation for fast contents discovery, trailer gen- eration, and minimally disruptive video-ads insertion. In addition, our proposed boundary-aware self-supervised learning algorithm can be applied to learn contextual representation of untrimmed, long videos. This would drive future research in videos to move towards leveraging the extremely large number of raw videos on the web learn video representations. However, as negative effect of this, it would consume large amount of computational resources. Also, as any machine learning algorithm is highly likely be biased for training data, the learned representation would be biased and lead to an issue related to discrimination because the training data for self-supervised learning often not validated by humans.
Reproducibility statement To make our algorithm reproducible, we present implementation de- tails in Section[4. of the main paper and in Section[Boo the supplementary material; we also provide a pseudocode for DTW-based pseudo-boundary discovery method. addition, we will release the code and parameters of trained models.
REFERENCES
Unaiza Ahsan, Chen Sun, and Irfan Essa. DiscrimNet: Sunii-upeerieed Action Recognition from Videos using Generative Adversarial Networks. arXiv preprint 101.021/00 2018.
Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. A Deep Siamese Network for Scene De- tection Broadcast Videos. Proceedings of the 23rd ACM international conference on Multi- media, 2015.
Donald Berndt and James Clifford. Using Dynamic Time Warping to Find Patterns in Time Series. In KDD workshop, 1994.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised Learning of Visual Features by Contrasting Cluster Assignments. arXiv preprint 1xXiv:2006.09882 2020.
Chien-Yi Chang, De-An Huang, Yanan Sui, Li Fei-Fei, and Juan Carlos Niebles. D3TW: Dis- criminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation. In CVPR, 2019.
Vasileios T Chasanis, Aristidis Likas, and Nikolaos P Galatsanos. Scene Detection in Videos using Shot Clustering and Sequence Alignment. IEEE transactions on multimedia, 11(1):89-100 2008.
Shixing Chen, Xiaohan Nie, David Fan, Dongqing Zhang, Vimal Bhat, and Raffay Hamid. Shot Contrastive Self-Supervisee Learning for Scene Boundary Detection. In CVPR, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A Simple Framework for Contrastive Learning of Visual Representations. In ICML, 2020a.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. UNITER: UNiversal Image- -Ext Representation Learning. In ECCV, 2020b.
Ishan Dave, Rohit Gupta, Mamshad Nayeem Rizve, and Mubarak Shah. TCLR: Temporal Con- trastive Learning for Video Representation. arXiv preprint arXiv:2101.07974, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-scale Hierarchical Image Database. In CVPR, 2009.
Yazan Abu Farha and Jurgen Gall. MS-TCN: Multi-Stage Temporal Convolutional Network for Action Segmentation. In CVPR, 2019.
Christoph Feichtenhofer. Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. Large-Scale Study Unsupervised Spatiotemporal Representation Learning. In CVPR, 2021.
10

---

Under review as a conference paper at ICLR 2022
Daniel Fried, Jean-Baptie Alayrac, Phil Blunsom, Chris Dyer, Stephen Clark, and Aida Ne- matzadeh. Learning to Segment Actions from Observation and Narration. arXiv preprint arXiv:2005.03684, 2020.
Bo Han and Weiguo Wu. Video Scene Segmentation using A Novel Boundary Evaluation Criterion and Dynamic Programming. In IEEE International conference multimedia and expo, 2011.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum Contrast for Unsupervised Visual Representation Learning. In CVPR, 2020.
Dan Hendrycks and Kevin Gimpel. Gaussian Error Linear Units (GELUs). arXiv preprint arXiv:1606.084 08415, 2016.
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. MovieNet: A Holistic Dataset for Movie Understanding. In ECCV, 2020.
Longlong Jing and Yingli Tian. Self-Supervised Spatiotemporal Feature Learning by Video Geo- metric Transformations. arXiv preprint arXiv: 1387, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015.
Hilde Kuehne, Alexander Richard, and Juergen Gall. A hybrid RNN-HMM Approach for Weakly Supervised Temporal Action Segmentation. IEEE transactions on pattern analysis and machine intelligence, 42(4):765-779, 2018.
Anna Kukleva, Hilde Kuchne, Fadime Sener, and Jurgen Gall. Unsupervised Learning of Action Classes with Continuous Temporal Embedding. CVPR, 2019.
Sateesh Kumar, Sanjay Haresh, Awais Ahmed, Andrey Konin, M Zeeshan Zia, and Quoc-Huy Tran. Unsupervised Activity Segmentation by Joint Representation Learning and Online Clustering. arXiv preprint arXiv:2105. 12220005..3333 13353, 2021.
Colin Lea, Austin Reiter, René Vidal, and Gregory D Hager. Segmental Spatiotemporal CNNs for Fine-Grained Action Segmentation. In ECCV, 2016.
Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang. Unsupervised Representa- tion Learning by Sorting Sequences. In ICCV, 2017.
Jun Li and Sinisa Todorovic. Set-Constrained Viterbi for Superuper Action Segmentation. In CVPR, 2020.
Jun Li and Sinisa Todorovic. Action Shuffle Alternating Learning for Unsupervised Action Seg- mentation. In CVPR, 2021.
Jun Li. Peng Lei, and Sinisa Todorovic. Weakly Supervised Energy-based Learning for Action Segmentation. ICCV, 2019.
Linjie Li, Jie Lei, Zhe Gan, Licheng Yu, Yen-Chun Chen, Rohit Pillai, Yu Cheng, Luowei Zhou, Xin Eric Wang, William Yang Wang, et al. VALUE: Multi-Task Benchmark for Video-and- Language Understanding Evaluation. In NeurlPS, 2021.
Chao Liang, Yifan Zhang, Jian Cheng, Changsheng Xu, and Hanqing Lu. A Novel Role-Based Movie Scene Segmentation Method. In Pacific-Rim Conference on Multimedia, pp. 917-922. Springer, 2009.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and Learn: Unsupervised Learning using Temporal Order Verification. In ECCV. 2016.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Pre- dictive Coding. arXiv preprint arXiv: 1807 03740.448 2018.

---

Under review as a conference paper at ICLR 2022
Rui Qian, Tianjian Meng, Boqing Gong, Ming-Hsuan Yang, Huisheng Wang. Serge Belongie, and Yin Cui. Spatiotemporal Contrastive Video Representation Learning. In CVPR, 2021.
Anyi Rao, Linning Xu, Yu Xiong, Guodong Xu, Qingqiu Huang, Bolei Zhou, and Dahua Lin. A Local-to-Global Approach to Multi-modal Movie Scene Segmentation. In CVPR, 2020.
Zeeshan Rasheed and Mubarak Shah. Scene Detection in Hollywood Movies and TV Shows. In CVPR, 2003.
Zeeshan Rasheed and Mubarak Shah. Detection and Representation of Scenes in Videos. IEEE transactions on Multimedia, 7(6):1097-1108 2005.
Byungseok Roh, Wuhyun Shin, Ildoo Kim, and Sungwoong Kim. Spatially Consistent Representa- tion Learning. In CVPR, 2021.
Daniel Rotman, Dror Porat, and Gal Ashour. Robust and efficient video scene detection using optimal sequential grouping. In 2016 IEEE international symposium on multimedia (ISM), 2016.
Daniel Rotman, Dror Porat, and Gal Ashour. Optimal Sequential Grouping for Robust Video Scene Detection using Multiple Modalities. International Journal of Semantic Computing, 11(02):193- 208, 2017.
Yong Rui, Thomas S Huang, and Sharad Mehrotra. Exploring Video Structure beyond The Shots. In Proceedings. IEEE International Conference on Multimedia Computing and Systems (Cat. No. 98TB100241), 1998.
Yuhan Shen, Lu Wang, and Ehsan Elhamifar. Learning To Segment Actions From Visual and Lan- guage Instructions via Differentiable Weak Sequence Alignment. In CVPR, 2021.
Panagiotis Sidiropoulos, Vasileios Mezaris, Ioannis Kompatsiaris, Hugo Meinedo, Miguel Bugalho, and Isabel Trancoso. Temporal Video Segmentation to Scenes using High-level Audiovisual Fea- tures. IEEE Transactions Circuits and Systems for Video Technology, (8))116631177) 2011.
Yaser Souri, Mohsen Fayyaz, Luca Minciullo, Gianpiero Francesca, and Juergen Gall. Fast Weakly Supervised Action Segmentation using Mutual Consistency. IEEE Transactions on Pattern Anal- ysis and Machine Intelligence, 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. The journal of machine learning research, 15(1):1929-1958, 2014.
Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov. Unsupervised Learning of Video Representations using LSTMs. In ICML, 2015.
Chen Sun, Fabien Baradel, Kevin Murphy, and Cordelia Schmid. Learning Video Representations using Contrastive Bidirectional Transformer. arXiv preprint 1906:099000573333 2019a.
Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. VideoBERT: A Joint Model for Video and Language Representation Learning. In ICCV, 2019b.
Makarand Tapaswi, Martin Bauml, and Rainer Stiefelhagen. StoryGraphs: Visualizing Character
Interactions as a Timeline. In CVPR, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NIPS, 2017.
Rosaura G VidalMata, Walter J Scheirer, Anna Kukleva, David Cox, and Hilde Kuehne. Joint Visual-Temporal Embedding for Unsupervised Learning of Actions Untrimmed Sequences. In WACV,202
Carl Vondrick. Hamed Pirsiavash, and Antonio Torralba. Generating Vidcos with Scene Dynamics. NIPS, 2016.
Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Track- ing Emerges by Colorizing Videos. In ECCV, 2018.
12

---

Under review as a conference paper at ICLR 2022
Zhe Wang, Hao Chen, Xinyu Li, Chunhui Liu, Yuanjun Xiong, Joseph Tighe, and Charless Fowlkes. Unsupervised Action Segmentation with Self-supervised Feature Learning and Co-occurrence Parsing. arXiv preprint arXiv:2105.14155: 2021.
Dejing Xu, Jun Xiao, Zhou Zhao, Jian Shao, Di Xie, and Yueting Zhuang. Self-Supervieed Spa- tiotemporal Learning via Video Clip Order Prediction. In CVPR, 2019.
Mengmeng Xu, Juan-Manue Pérez-Rúa, Victor Escorcia, Brais Martinez, Xiatian Zhu, Li Zhang, Bernard Ghanem, and Tao Xiang. Noundary-sensitive Pre-training for Temporal Localization in Videos. arXiv preprint 17xxv:2011.108321 2020.
Yang You, Igor Gitman, and Boris Ginsburg. Large Batch Training of Convolutional Networks. arXiv preprint arXiv: 700.07088...... 2017.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion Image Database for Scene Recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2017.
Dimitri Zhukov, Jean- Baptiste Alayrac, Ramazan Gokberk Cinbis, David Fouhey, Ivan Laptev, and Josef Sivie. Cross- Task Weakly Supervised Learning from Instructional Videos. In CVPR, 2019.
13

---

Under review as a conference paper at ICLR 2022
A DATASET STATISTICS
Table shows the data statistics of different video scene segmentation datasets. We found the limited number of datasets that provide the scene boundary annotations and, as far as we know, the MovieNettSSeg (Huang et al., 2020) is the largest-scale video scene segmentation dataset.
Table 5: Comparison of existing video scene segmentation datasets. Note that we brought the table from (Rao et al. 2020) with an update on the MovieNet-SSeg dataset.
Dataset
#Video
#Scene
#Shot
Time (h)
Source
BBC (Baraldi al... 2015
11
670
4.9K
Documentary
OVSD (Romman 2016
300
10K
10
MiniFilm
MovieNet-SSeg (Huang al..
2020
318
42K
500K
Movies
MovieNet (Huang et al. 2020
1,100
1.6M
Movics
B ADDITIONAL IMPLEMENTATION DETAILS
Additional details of the shot encoder (i.e., ResNet-50) and the contextual relation network (i.e., Transformer) are follows. For Transformer, the dimensions are set to (L = 2, H1 768, A = 8) where L, H and mean the number of stacked transformer blocks, the dimension of hidden ac- tivation and the number of attention heads, respectively. We apply the Dropout technique (Sri- vastava et al. 2014) on hidden states and attention weights with a probability of 10% and use GELU (Hendrycks & Gimpel| 2016) as an activation function. For the shot encoder, each shot is given by three key-frames (i.e., Ne = 3) and a shot encoding is computed by the averaged feature after inferring individual three key-frames by ResNet-50; note that, speed up the training, we use randomly sample one key-frame out of three during the pre-training.
For data augmentation of key-frames in a shot, we adopt PyTorch's torchvision package. Given sequence of shots, we apply random crop (with resize), random flip, random color jitter and random Gaussian blur. In detail, firstly, the cropping is performed with a random size (i.e., scales between [0.14, 1.0] of the original size) and random aspect ratio (between 3/4 to 4/3), and then the cropped one is resized to (224,224). Secondly, we apply a random horizontal flip with a probability of 50%. Thirdly, as color augmentation, we perform a random color jitter (with a probability of 80%) and random color dropping to gray scale (with probability of 20%). The color strength parameters for jittering are set to (brightness, contrast, saturation, hue} (0.2, 0.2, 0.2, 0.05). Finally, Gaussian blur applied with a probability of 50% where a standard-devvation of spatial kernel is set to 10. 2.0]. Note that the same augmentations shots while different color jittering is applied on individual shots. Also, for Solow different augmentation compared that applied on S.,
applied to all key-frames in the input sequence S,, of
we perform a n
During the pre-training stage, the model parameters are randomly initialized and then trained using the proposed pretext tasks. We use LARS (You al. (017) learn the model (except for parameters of bias and narch-Normalizattion) with a mini-batch of 256 shot sequences, a base learning rate of 0.3, momentum of 0.9, weight decay of 10-6 6 and trust coefficient of 0.001. We pre-train the model for 10 epochs with a linear warm-up strategy for epoch followed by learning rate decaying with cosine schedule. The temperature
al.
Eq. (5) set to 0.1.
In the fine-tuning stage, we initialize the parameters of the shot encoder and the contextual relation network by that of the pre- treitrained ones, however, we freeze the parameters of the shot encoder following Chen et al. (2021). We fine-tune the contextual relation network and the scene boundary detection head for 20 epochs using Adam (Kingma & Ba 2015) with learning rate of 10-5 and mini-batch of 1024 training examples. The learning rate is decayed with cosine schedule without a warm-up stage.
C COMPARISON WITH SHOT-LEVEL SELF-SUPERVISED LEARNING
As mentioned in the main paper, our approach is distinguishable from the shot-level pre-training approach (Chen et al. 2021 Qian al. 2021) in that the objectives used in approach (BaSSL.)
14

---

Under review as a conference paper at ICLR 2022
SSM CGM PP MSM
Shot Contrastive Loss
Boundary (1)? or not (o)?
Boundary (1)? or not (0)?
Pseudo boundary Disrovery
CRN (fru)
CRN (FeN)
CRN (Fenn
transfer
transfer
Shot Encoder Unal
Shot Enzoder Unuel
Shot Encoder Unel
Shot Encoder Unel
query shot
similar
000000000 Fine tunintnnnnnggnn Stage
00000000
shot
Shot-level Pre-training
Boundary-aware Pre-training
Fine tiniin Stage
(a) Existing Approach
(b) Our Approach
Figure 5: Comparison between existing approaches and ours for video scene segmentation. The existing approach focuses only on learning shot-level representation given by shot encoder (fENc). In contrast, our boundary-aware pre-training method focuses on learning contextual representation by taking neighbor shots into account. Thus, our method can learn both the shot encoder (fENc) and the contextual relation network (CRN; fCRN) and transfer their parameters during the fine-tuning stage.
Table Comparison between the shot-level pre-training and the proposed pre-training approaches.
Check List
Shot-level Pre-training
Boundary- Pre-training
Network architecture
JENC
fENc + SCRN
Training input
a pair of shots (#shots: 2) a sequence of shots (#shots: 2K+1) yes
Weights transferable for fENc?
yes
Weights transferable for fCRN?
no shot-shot
yes shot-scene
Positive pair in contrastive learning
is to learn contextual representations by taking neighbor shots into account. Figure 5 provides clear summary of comparison between shot-level pre-training and our BaSSL. Firstly, shot-level pre-training takes pair of two shots as input while BaSSL takes sequence of shots. Secondly, shot-level pre-training aims to train shot encoder (fENC) only, while BaSSL trains both the shot en- coder and the contextual relation network (fENC and fCRN). In contrast to the shot-level pre-training that requires to train feRN from scratch during the fine-tuning stage, BaSSL benefits from weight transfer by pre-training the parameters of SCRN with large-scale in-domain data in advance. Note that the results (M6-7) in Table 2 show that the weight transfer of sCRN is important to improve the video scene segmentation performance. Finally, the contrastive learning objective in shot-level pre-training locates the representations of two shots (query and positive) to be close to each other, whereas Shot-Scene Matching objective in our approach performs the same task but with a shot (query) and its associated scene (positive; a sequence of shots). The Table T summarizes the afore- mentioned comparisons.
D ALGORITHM FOR PSEUDO-BOUNDAL DISCOVERY
In this section, we describe the details of pseudo-boundary discovery method applying DTW on S,, and Sslow, In practice, S, is given as a mini-batch resulting in a tensor with a shape of (B, S, NR. C, H, W) where individuals mean the batch size, the number of shots in Sn (i.e., 2K + 1), the number of key-frames in a shot, channels, frame height and frame width, respectively. Then, obtain Sylow E BBx2x H that is composed of the first and last shots in S,,. We apply two different augmentation functions into key-frames in S,, and Sylow, respectively. Next, compute encoded representation of S,, and Sslow using fENC- Note that during the pre-training stage, we randomly sample one key-frame among N& candidates in shot and then reshape the input tensor as (B*5, C, H. W) or (B*2, C, H. W) to be forwarded by the shot encoder fENc; thus the tensor shape of the encoded shot representation is given by (B, S. D.) or (B, 2, D.) after apply reshaping, where D. means the dimension of encoded feature. Finally, given two sequences of encoded representation for S, and Sllw DTW provides two sub-sequences Sieft and stight and a pseudo boundary shot Sn+b.. The algorithm Tillustrates the details. In addition, to demonstrate the simplicity of the alignment computation using DTW, we include the PyTorch code in Listing T The implementation of DTW can be done in 5 lines of python code using tslearn package.
15

---

Under review as a conference paper at ICLR 2022
from tslearn import metrics import numpy as np def
seql,, 2)q
Input:
seq 1: sparse shots embedding, shape torch. (12, dimi)
seq_2: 2: dense shots embedding, shape torch.Size (IM, dimi), >
Output:
dtk_path: output of DTW algorithm, shape torch. ([N, diml)
"**
cost (1-torch. bmm (seg_l, seq 2, transpose (1, 2))) .ummpy () dtw paw_ = I]
for bsz in zange (cost. shape [0])
path, = metrics. dtw hettw_pahh_freee mottricctcccc [baz], matric="precompptes
atw_path. append (np. (n asarray (_pah))) $ torch.Size (IM, dim))
return dtw_path
Listing 1: PyTorch code for alignment computation using DTW given two sequences. The tslearn
package is used for DTW path calculation.
Algorithm DTW-based pseudo- discovery
1: Input: Shot encoder fene. contextual relation network fCRN, and an input shot sequence Sn, (Sn-K,...," Sn: .... k centered at n shot 8, with neighbor size K, two image augmentation functions \ 2
aug
2: (En, Eslow) ) (1), ()
3: for in K to n + K do
4: e; + fENC (E (s:)
extract shot-level representations for all shots
5: E,, (Enjes)
append
6: end for
7: for i in (n Knn + K} do
8: e: - --JENC( (> (s:)) // extract shot-level representations for slow sequence
9: Eslow - (Exlow e;} // append
10: end for
Sleft Sright b' + DTW(E,, EA!"") apply dynamic time warping
12: Output: Two continuous ovnrraealgpinn sub-sequences Sleft and Stight and a pseudo boundary shot S
E RESULTS ON ADDITIONAL DATASETS
We further compare BaSSL with shot-level pre-training baselines on additional two BB and OVSD. Note that the train and test splits are not available and the dataset size is extremely limited (11 and 21 videos in BBC and OVSD, respectively); in addition, 2 out 21 videos in OVSD not available. Thus, infer predictions using models trained MovieNet-SSeg without fine- tuning on BBC and OVSD. The results are summarized in Table The result shows the superiority of our method compared shot-level pre-training baselines.
Table 7: Comparison between our method and shot-level pre- pre-training baselines on BBC and OVSD datasets. The numbers mean AP.
Model
SimCLR (instance)
SimCLR (temporal)
SimCLR (NN)
BaSSL
BBC
32.34
34.18
32.92
39.98
OVSD
25,45
24.92
25.02
28.68
16

---

Under review as a conference paper at ICLR 2022
Table 8: Scene clustering quality measured by normalized mutual information (NMI) metric.
Scene Length
Model
A +. (Short + Long)
Short (N.=8)
Medium (N,=16)
Long (N,=32)
ImageNet SimCLR (temporal)
67.50
61.60
56.25
-16.67%
82.40
81.65
78.99
-4.14%
SimCLR (NN)
83.54
83.17
81.25
-2.75%
BaSSL (ours)
86.22
86.72
85.63
-0.68%
Table 9: Ablation study on the combination of boundary-awand pretext tasks measured by NMI.
Pretext Tasks
NMI
Gain (A%)
SSM
85,45
0.00%
SSM+MSM
85.64
+0. +0.19%
SSM+MSM+CGM
85.93
+0.33%
SSM-MSM+CGM-PP
86.71
+0.91%
F MEASURING REPRESENTATION QUALITY AT PRE-TRAINING STAGE
The normalized mutual information (NMI) is a metric for clustering algorithms (e.g., K-Means), which measures the clustering quality. Since clustering with good representations forms clear boundaries between different classes, NMI can be considered as a proxy to measure the quality of our pre-trained models. Specifically, we randomly sample 100 scenes from the test split of MovieNet-SSeee while we vary the length of scenes N E {8, 16, 32}. Then, we perform K-Means clustering on N₂ x 100 shot representations extracted by the pre-trained model with the number of classes K=100. We intend to form single cluster for each scene in this formulation, assuming that high-quality representation for movic scene segmentation would locate the shot features within the same scene close to each other. Considering the randomness in the K-Means clustering and scene sampling, we report the averaged score from five trials.
In Table our SimCLK version implementation of ShotCoL. The result shows that BaSSL outperforms the shot-level pre-training baselines and the model pre-trained using ImageNet dataset. With respect to different scene lengths (Ne; the number of shots included in a single scene), we found our BaSSI. is more robust than the other baselines. Since the visual diversity across the shots increases as the scenes become longer (N,=8 + 32), it is natural that the NMI score for each baseline is degraded. However, is remarkable that, by increasing the number of shots from 8 to 32, the performance of BaSSL drops only -0.68% while the other baselines suffer from severe degradation. This demon- strates the effectiveness of BaSSL in maximizing mntra-ccene similarity.
compare the NMI score between different pre-trained models; SimCLR (NN) is
In addition, we perform ablation study of our algorithm by adding pretext tasks one by one, and mea- sure the corresponding NMI scores. The result in Table shows that better NMI score is achieved as more pretext tasks are combined together. This tendency is also observed in our ablation in Table which indicates the NMI score of pre-trained models is highly correlated with the final performance after the fine-tuning
G QUALITATIVE ANALYSIS
Visualization of similarities between consecutive shots To qualitatively check the effect of indi- vidual pretext tasks, we visualize the matrix of cosine similarity between shot representations from the randomly sampled 16 consecutive shots in Figure 6 The shot representations are computed by models without the fine-tuning in order to solely focus the behavior of each objective at the pretraining stage. When the MSM is used only, approximately three clusterings are shown, but sim- ilarity around boundaries is smoothed. Next, when we add PP, dissimilarities around the boundaries are to be sharpened. Then, with additional CGM, the clusters are more clearly obtained. Finally, adding SSM makes the similarity of shots within the same cluster higher (i.e., more yellow ones).
We compare the quality of discovered pseudo- boundaries with the ground truth scene boundaries in Figure 7 In most cases, we observe the beuuda-boundaris identified by
Sheudo-boundaries
17

---

Under review as a conference paper at ICLR 2022
Input Shots
80
5 (d) MSM PP CGM SSM
(a) MSM
(b) MSM PP
(c) MSM CGM
Figure 6: Visualization of similarity (below) between shot representations in randomly sampled consecutive shots (above). We observe that the shot representations are clearly clustered as adding pretext tasks one by one.
the DTW algorithm are successfully located in close distance with the ground truth ones. This result validates idea considering the problem of discovering useudo-boundary as temporal alignment problem between two sequences with different frequencies (S,, and Sslow).. At the same time, we illustrate the failure cases. Although discovered pseudo-boundary does not match the ground truth in this case, we figure the determined boundary is not always arbitrary. For example, the mismatch often caused by the noise existing in the ground truth (see the first row in the failure cases). On the other hand, in case all shots are visually similar (see the third row in the failure cases), the DTW solely relying on the visual modality fails to find the correct boundary.
Predicted scene boundaries The figure S illustrates the scene boundary predictions of differ- ent models. Comparing with the baselines, we observe that our approach, BaSSL, shows qualita- tively better performance for video scene segmentation. On the other hand, we observe the over- segmentation issue in many cases using any compared methods (including ours). Our finding implies that achieving the highest recall only does not guarantee the highest performance in practice. We reckon that further studies on this over-segmentation problem would be a highly important topic when it comes to real- world application.
18

---

Under review as a conference paper at ICLR 2022
Success cases of DTW
10
10
(b) Failure cases DTW
AUL
10
JAG
GT
DnW
Comparison between the ground truth scene boundaries and the discovered pseudo- boundaries based on the DTW algorithm. The examples are sampled from the MovieNet-SSee. dataset. All boundary shots are highlighted in red.
Figure 7:
19

---

Under review as a conference paper at ICLR 2022
GT
Imagataal
Cun ShatCal
GT
Curs ShoiCal ImagaNNet
GT
Our ShacCal Imaget
GT
Oaaa ShorCal Imsegebaat
GT
Oira ShoCCll ImagieNet
GT
Dura ShoCCol Imagelet
Figure 8: Comparison of boundary detection results from three pre-training approaches: ImageNet pre-trained ResNet, ShotCoL, and BaSSL. The first row shows the reference that is composed of two adjacent scenes divided by the ground truth boundary. We visualize the shots that are assigned to the same scene segments with the same colored border.
20


Reviews:
[{"review_id": "ojRKSsj370P", "reviewer": "Reviewer_DwYy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "(+) This paper studies an interesting problem that make a video network aware of boundary infomation for video scene segmentation. This is a general problem in many video understanding tasks such as temporal action localization, video language grounding.\n\n(+) The proposed method is under self-supervised learning settings, which does not require any extra annotation cost, but still improve the video network for the down stream task.\n\n(+) The discovery of pseudo boudnarys follow the intuition of the scene segmentation task. Moreover, the algorithm garantees that it can alway generate one unique pseudo boundary from a given video.\n\n(+) According to their experimental results, the proposed method can achieve a new state-of-the-art on the MovieNet-SSeg benchmark.\n\n(-) The core idea of this paper is similar to BSP (Xu et al., 2020), which generate psuedo boundaries and devolped pretext tasks on them. From Table 4, the performance of this work is only merginally higher (1.64%) than BSP.\n\n(-) Not sure if or not DTW is necessary to do the temporal alignment problem. Since the second sequence has only 2 elements, the solution space is only O(K). The optima could be efficiently found by an argmanx operation after metrix multiplication. \n\n(-) In Table 3, the last column 'Sum' seems to be the summation. This does not make sense to me to sum different metrics together without further explaination. \n", "summary_of_the_paper": "This paper introduces a self-supervised learning method for the video scene segmentation task, which needs to temporally localize scene boundaries in a video. Their proposed method defines pseudo-boundary from a sequence of shots by splitting it into two sub-sequences and train a model to predict the pseudo-boundary with three novel tasks: Shot-Scene Matching (SSM), Contextual Group Matching (CGM), and Pseudo-boundary Prediction (PP). Experiments on pre-training and transfer learning show that this method is critical to improving the video scene segmentation performance, and it achieves the new state-of-the-art on the MovieNet-SSeg benchmark.", "summary_of_the_review": "In short, this paper studies an interesting problem, and proposes an convinient and effective method as a solution. Although the proposed solution is not very novel, and not strongly techniqual, it is marginally above the acceptance threshold for the potential impact beyond video scene segmentation. \n\nI also would like to see the opinion from other reviewers. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VMg5IRlgEwg", "reviewer": "Reviewer_pa9p", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My most concern is about pseudo-boundary discovery. As the rest loss functions are borrowed from current self-supervised loss functions. First question is: why only use one boundary in each video? This operation is not reasonable as the video may contain multiple scenes. Also, how to define the different scenes? For figure 4 (c), we can see the shots come from the truck inside, track outside and inside again. So, the generated pseudo-scene B is confused as it contains another scene. This task is totally different from action segmentation as action segmentation has clear boundaries in each video. In my opinion, without the so-called pseudo-boundary discovery, even video-level loss for self-supervised video representation, such as [1,2] can train a strong backbone for second stage training.\n[1] Self-supervised Co-training for Video Representation Learning, neurIps\n[2] Space-Time Correspondence as a Contrastive Random Walk,NeurIps\n\nMy second concern is about the experimental result. The proposed pre-training the main contribution of the whole paper with complex loss functions design. However, from table 1, we can see that without finetune, the proposed method is inferior to the old Grouping method. The main performance promotion(31.55->56.26) comes from the second stage. \nFinally, from the ablation studies, the whole paper is a combinatoric paper, which combines many existing ideas instead of proposing novel ones. The ablation studies do not give much insight into the method or behavior of the proposed loss. Especially compared to existing work ShotCoL, the performance improvement is limited but the methods are much complicated. \n\nSome other suggestions for improvement: 1. One dataset is not enough for a top-conference work, more datasets, such as AdCuepoints should be evaluated. 2. Some typos: 1) % p.  2) . For fair comparison, all pre-training methods employ ResNet-50 (He et al., 2016) as the shot encoder fENC and we pre-train the models for 10 epochs.", "summary_of_the_paper": "The paper introduces a boundary-aware self-supervised learning framework for video scene segmentation. Based on the contrastive learning protocol, the authors employ DTW to generate dimidiate video segmentation from unlabeled video. Using these segments, the authors employ four loss functions to pre-train a backbone network.", "summary_of_the_review": "The quality of the whole manuscript should be further improved. The motivation of the whole task is not clear, especially compare to action segmentation task. May raise the score if author can handle all my concerns. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "AX-KLNxeEfX", "reviewer": "Reviewer_3Yuj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- This paper is clear and easy to follow.\n- The overall framework looks reasonable and effective.\n- The proposed method achieves good results and outperforms previous methods.\n\nCons\n- The proposed framework is not generic and hard to be applied to other methods for the task.\n- In the pseudo-boundary discovery stage, the given video sequence is separated into two scenes, which is reasonable if there the input contains only two scenes. Yet, one may wonder that how the training data is prepared and what if there are more than two scenes. Given the high performance reported in experiments, the related analysis is important which however is missing.\n- From Tab.3, the effectiveness of MSM is marginal.\n- The lack of insights makes this paper less interesting. E.g. the authors claim that SSM and CGM help to optimize inter-/intra- feature distribution, yet lack direct evidence/demonstration. Simply reporting some improvements in final accuracy is less convincing, as the performance can be affected by other reasons like stronger network architecture, better training protocol, etc.", "summary_of_the_paper": "This paper presents a boundary-aware self-supervised learning framework for the task of video scene grouping.  During the pre-train stage, the authors first apply DTW to find a pseudo boundary in a given video sequence, and then applies Shot-Scene Matching and Contextual Group Matching to optimize the intra-scene and inter-scene distribution of features. Moreover, Pseudo-boundary Prediction and Masked Shot Matching are utilized to help the model identify transitional moments. ", "summary_of_the_review": "This paper presents an effective framework for the task and reports good performance. Yet I am still concerned about its lack of insights and generalizing ability.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on video scene segmentation.  The reviewers appreciated the introduction of a boundary-aware pre-training method.  However, concerns were raised regarding limited novelty, empirical effectiveness, and generic applicability.  The reviewers engaged in significant discussion based on the other reviews and authors' responses.  Based on these discussions the reviewers concluded that while the proposed method does have differences with respect to BSP, the overall contributions were not sufficient for inclusion in ICLR.
Decision: Reject