Boundary-aware Pre-training for Video Scene Segmentation

Abstract: Self-supervised learning has drawn attention through its effectiveness in learning in-domain representations with no ground-truth annotations; in particular, it is shown that properly designed pretext tasks (e.g., contrastive prediction task) bring significant performance gains for a downstream task (e.g., classification task). Inspired from this, we tackle video scene segmentation, which is a task of temporally localizing scene boundaries in a video, with a self-supervised learning framework where we mainly focus on designing effective pretext tasks. In our framework, we discover a pseudo-boundary from a sequence of shots by splitting it into two continuous, non-overlapping sub-sequences and leverage the pseudo-boundary to facilitate the pre-training. Based on this, we introduce three novel boundary-aware pretext tasks: 1) Shot-Scene Matching (SSM), 2) Contextual Group Matching (CGM) and 3) Pseudo-boundary Prediction (PP); SSM and CGM guide the model to maximize intra-scene similarity and inter-scene discrimination while PP encourages the model to identify transitional moments. Through comprehensive analysis, we empirically show that pre-training and transferring contextual representation are both critical to improving the video scene segmentation performance. Lastly, we achieve the new state-of-the-art on the MovieNet-SSeg benchmark. The code will be released.

Reviews and author responses:

 > (1) For task clarification, the authors claim that 'However, we emphasize that action segmentation mainly focuses on identifying human behaviors (especially, actions) while scene segmentation aims at capturing more compositional semantics (e.g., composition of multiple actions). This difference makes application of ideas or algorithms in action segmentation to scene segmentation difficult.' From this view, scene segmentation can be regarded as coarse action segmentation and we can use the action segmentation to implement scene segmentation easily. So, what is the meaning of scene segmentation? 

**Response:**
We first clarify the meaning of scene segmentation to address the concerns about “the motivation of the video scene segmentation is not clear especially compared to the action segmentation task”. The term scene is widely used in filmmaking, where the scene is considered as a basic unit for understanding the story of movies; video scene segmentation can be thought of as a task of dividing a video into segments telling a sub-story. Therefore, the main purpose (or motivation) and target video domains are different in two tasks; action segmentation aims to **recognize** human behavior (i.e., action) in activity videos such as **cooking activity**, while, as a pre-processing step, scene segmentation is desired to **divide** a long video (such as **a movie**) into semantic segments.  
We understand why the reviewer claims that our task can be seen as a coarse action segmentation. However, we strongly argue that there still exists a clear gap between them. As mentioned in LGSS [A], semantic diversity and compositionality of scene makes it impossible to define the scene categories (for sub-story) even at a coarse level. This is the main reason that the video scene segmentation algorithms (LGSS [A], ShotCoL[B] and ours) formulates the task as a binary classification of whether each shot is boundary or not, while action segmentation algorithms formulates the task as a multi-class classification. Such a difference makes the application of action segmentation algorithms to scene segmentation difficult.

[A] Anyi Rao, et al., A Local-to-Global Approach to Multi-modal Movie Scene Segmentation, CVPR, 2020  
[B] Chen, Shixing, et al., Shot Contrastive Self-Supervised Learning for Scene Boundary Detection, CVPR, 2021

---
> (2) For the presented method, especially boundary generation which is the main contribution of the whole manuscript, I strongly agree with Reviewer DwYy about the limited novelty. The boundary generation process is similar to BSP while the loss function is borrowed from current self-supervised loss directly.

**Response:**
As the reviewer 3Yuj recognized, our pseudo-boundary **discovery** process is different from the pseudo-boundary **generation** process of BSP, and we demonstrated our method is more effective than that of BSP. Also note that many important works in CV fields also use widely-used loss functions (or existing ideas) for their purpose  rather than always proposing new ones. More importantly, without the idea of identifying pseudo-boundaries from an input shot sequence, it is difficult (or even impossible) to apply the “current self-supervised loss” for boundary-aware pre-training. Finally, we would like to emphasize that, so far as we know, our work is the first to introduce boundary-aware pre-training in video scene segmentation (and even including action segmentation).


 Thanks for the responses. The authors' explanation about multi-scene training samples looks reasonable to me. 
I agree with peer reviewers' concern about the novelty of loss functions, yet I would like to slightly argue for the boundary-aware learning of this work.  
As far as I understand, BSP (Xu et al) synthesize boundaries with ground-truth knowledge, while this work explores unsupervised pseudo-boundary and demonstrates its effectiveness with good performance, which are different contributions in my opinion.
Therefore, I tend to maintain the initial score.


 I have read the comments from other reviewers as well as the authors' responses. I still concerned about the proposed video scene segmentation task as well as the proposed method. (1) For task clarification, the authors claim that 'However, we emphasize that action segmentation mainly focuses on identifying human behaviors (especially, actions) while scene segmentation aims at capturing more compositional semantics (e.g., composition of multiple actions). This difference makes application of ideas or algorithms in action segmentation to scene segmentation difficult.' From this view, scene segmentation can be regarded as coarse action segmentation and we can use the action segmentation to implement scene segmentation easily. So, what is the meaning of scene segmentation?
(2) For the presented method, especially boundary generation which is the main contribution of the whole manuscript, I strongly agree with Reviewer DwYy about the limited novelty. The boundary generation process is similar to BSP while the loss function is borrowed from current self-supervised loss directly. 

(3) Thanks for updating the experimental section by adding two small datasets in the manuscript.

Overall, I maintain the initial score.

 > Comment: (1) the boundary generation process is similar to BSP. I do agree that the boundary generation introduced in this paper is SSL but BSP is not. However, I did not see a big difference between them. The main idea of both works is to utilize pseudo boundaries. In implementation, either using SSL or not depends on the problem setup.

**Response:**
 We figure that if the method is SSL or not makes a huge difference; BaSSL (i.e., SSL) algorithm can be easily extended to large-scale data while BSP cannot do that due to the requirement for an expensive annotation process. We believe that such generalizability drives  a lot of attention on SSL approaches compared to supervised approaches recently.  
&nbsp;In addition, we also emphasize that **everything except for the idea of utilizing pseudo-boundary are totally different** between BSP and BaSSL. Let us think about the SSL algorithms in the image domain. SimCLR [A] and SwAV [B] would be considered to share the main idea of making positive pairs by data augmentation techniques (as like pseudo-boundary). However, their novelty comes from the design of pretext tasks. Similarly, the pretexts in BSP and BaSSL are totally different and we believe that such differences should be considered as our novelty compared with BSP.

[A] Ting Chen, et al., A Simple Framework for Contrastive Learning of Visual Representations, ICML, 2020  
[B] Mathilde Caron, et al., Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, NeurIPS, 2020

---
> Comment: (2) the loss functions are borrowed from current self-supervised loss functions. The author pointed out that the biggest contribution is the new framework. Then we need to know if the framework can be generalized to other tasks/datasets. Notably, the promising results on additional datasets are provided in the updated paper. This is a plus to me, but experiments on large scale datasets (in another similar task) should be more interesting.

**Response:**
As mentioned by the reviewer, it would have been better if we could provide results on large-scale datasets of another similar task (e.g., action segmentation). Unfortunately, we decided not to include it because of the following reasons. First, the datasets in action segmentation are composed of small-scale videos; the widely used three datasets (i.e., 50Salad [C], YouTube Instruction [D] and Breakfast [E]) contain 4.5 hour-long, 6 hour-long and 77 hour-long videos, respectively, while MovieNet is composed of 3K hour-long videos. Thus, we have believed that they are not suitable for self-supervised learning. Second, one recently introduced dataset (i.e., GEBD [F]) which collects generic boundaries (for such as sub-actions) by repurposing Kinetics dataset can be another option. However, the videos are too short (~10 seconds) and the boundaries divide too fine-level semantic chunks such as sub-actions; thus, the dataset is not suitable for our purpose of scene-level segmentation for high-level video understanding.  
&nbsp;While we still have a plan to apply our method on GEBD, we regret that the result would not be prepared in a short period of time for the remaining rebuttal. However, we will make sure to include the result for GEBD before the camera-ready.

[C] Sebastian Stein and Stephen J McKenna, Combining embedded accelerometers with computer vision for recognizing food preparation activities, In Proceedings of the 2013 ACM international joint conference on Pervasive and ubiquitous computing, 2013  
[D] Jean-Baptiste Alayrac, et al., Unsupervised learning from narrated instruction videos, CVPR, 2016  
[E] Hilde Kuehne, et al., The language of actions: Recovering the syntax and semantics of goal-directed human activities, CVPR, 2014  
[F] Mike Zheng Shou, et al., Generic Event Boundary Detection: A Benchmark for Event Segmentation, ICCV, 2021

---
> For comment (3) and (4)

**Response:** It is fortunate that our response (partly) addresses the reviewer’s concerns.


 Thanks for the detailed response! After reading the comments from reviewer pa9p and 3Yuj, my only concern of this paper is that the proposed idea is not very noval, and may hardly applied to other fields. 

(1) the boundary generation process is similar to BSP.
I do agree that the boundary generation introduced in this paper is SSL but BSP is not. However, I did not see a big difference between them. The main idea of both works is to utilize pseudo boundaries. In implementation, either using SSL or not depends on the problem setup. 

(2) the loss functions are borrowed from current self-supervised loss functions. 
The author pointed out that the biggest contribution is the new framework. Then we need to know if the framework can be generalized to other tasks/datasets. Notably, the promising results on additional datasets are provided in the updated paper. This is a plus to me, but experiments on large scale datasets (in another similar task) should be more interesting. 

Additional comments (less important):

(3) DTW for alignment may not be the best option, but it is acceptable to me due to the simple implementation and efficiency.

(4) Thanks for updating [A] and [B] in the manuscript.

 > Comment: (1) ... How to define a moment?
 
**Response:** As described in the Section 3.3, we find a pseudo-boundary by aligning intermediate shots either to the first or last shots in an input sequence. We adopt DTW due to its useful constraint on monotonicity for computing temporal alignment (please see our second response to Reviewer DwYy for more detail). Then, by solving Eq. (3), our method outputs two sub-sequences to be maximally distinct from each other; thus, “the strongest moment” can be obtained.
 
---
> Comment: (2) ... Please refer to action segmentation or action recognition[1] [1] Finegym: A hierarchical video dataset for fine-grained action understanding, cvpr
 
**Response:** If we understood correctly, the main reason the reviewer mentioned the action segmentation task is that the task aims to divide a video into multiple action segments adaptively similar to ours. However, we emphasize that action segmentation mainly focuses on identifying human behaviors (especially, actions) while scene segmentation aims at capturing more compositional semantics (e.g., composition of multiple actions). This difference makes application of ideas or algorithms in action segmentation to scene segmentation difficult.  
&nbsp;Regarding the FineGym paper the reviewer suggested, we find this work proposing a new dataset for the action recognition task. On top of the proposed dataset, the paper further tackles temporal action localization by finding a variable number of action instances; however, the action localization method requires localization labels, while our pseudo-boundary discovery is performed in an unsupervised setting. Thus, we believe that the idea or algorithm of FineGym is hard to be directly applied for the pseudo-boundary discovery process in an unsupervised setting.  

**For concerns about two scenes division**, what we claim here is that learning where is the strongest semantic change within a sequence of shots is sufficient to find **all** segments at testing. For a very long sequence with multiple segments, yes, as the reviewer concerns, our approach might learn only the strongest semantic change, missing other less obvious but still important changes. However, we train with its subsequences split into sufficiently short. Indeed, we search for a proper length of sequences as we mentioned in the experiment section. With length of sequences K=8, the majority of training input seqeuces actually has 2 or less scenes, so the reality is far from what the reviewer concerns. We may train with even shorter splits: now there might be no significant scene change in ground truth. Still, our approach tries to learn pseudo-boundary. Eventually, once trained on multiple sequences, the model is able to output reasonable estimation for each scene to be the last shot of a scene (that is, if there is a big gap compared to its next shot).  
&nbsp;In additoin, we might use an analogy to ranking problems. The main goal of ranking task is ordering all items in the sequence in the order of some criteria (e.g., relevance). At training, however, the ranker usually is trained to correctly order a pair of two items. Still, it eventually becomes able to sort more than 2 entries. Pair-wise training is more widely used and often outperforms list-wise approach (that is, directly optimizing the entire list in the ground truth), as it is effective as well as scalable. We hope this explanation and analogy help the reviewer to understand whether we find all pseudo-boundary at once or not does not really matter to solve this problem.  
 
---
> Comment: R2) ... However, from table 1, we can see that using more training epochs would lead to performance promotion easily (56.26->57.40). This is in totally conflict with the so-called extremely harder. These statements are too subjective.
 
**Response:** We agree that the statement would be subjective depending on the person. However, according to the Figure 1 in appendix of ShotCoL (https://arxiv.org/pdf/2104.13537.pdf), the ShotCoL is pre-trained for sufficiently long training epochs (>140 epochs, in contrast, 40 epochs for BaSSL). This means the score reported in ShotCoL paper may be obtained from the sufficiently converged model in their best optimization. Therefore, we think that the absolute improvement of +4%p from (presumably the best performance of) ShotCoL is not marginal considering the improvement of ShotCoL from LGSS (+6%p). Similar to the statement, we think “marginal” could also be subjective somehow depending on the person. Thus, we would like to respectfully ask why the reviewer thinks +4%p improvement from the previous state-of-the-art is still marginal and if any references on this can be kindly provided to us. We hope our reply can resolve the reviewer’s concern.


 We truly appreciate the reviewer for the constructive comments. Below are our responses to the major concerns about our paper.

### 1. Additional analysis for better insights
 We agree that it is important to provide more evidence or demonstration for better understanding of insights such as behaviors of individual pretext tasks. Thus, we conduct the following two additional analyses with pre-trained models:

(1) NMI-based clustering capability comparison  
 The normalized mutual information (NMI) is a metric for the quality of clustering. Since clustering with good representations forms clear boundaries between different classes, NMI can be considered as a proxy to measure the quality of our pre-trained models. In brief, we collect shots from randomly sampled 100 ground-truth scenes, then perform K-Means clustering over their shot representations and measure NMI. The result is updated in Appendix F, with a detailed process and our interpretation.

 In Table 8, we compare the NMI score between different pre-trained models. The result shows that BaSSL outperforms the pre-training baselines regardless of scene length. We would like to emphasize that BaSSL shows superior performance consistently with various scene lengths, while the competing models drop their performance significantly with longer inputs. This demonstrates the effectiveness of BaSSL in maximizing intra-scene similarity.

 On the other hand, we additionally report the NMI of our method with pretext tasks added one by one in Table 9. We observe the similar tendency from our ablation experiment in Table 3; better NMI score is achieved as more pretext tasks are combined together.

(2) Visualization of similarity between consecutive shots  
We qualititatively visualize the matrix of cosine similarity between shot representations from the sequence of shots (see Figure 6 in the updated draft). The shot representations are computed by models without fine-tuning, in order to solely focus on the behavior of each objective at the pretraining stage. When the MSM is used alone, approximately 3 clusters are shown, but similarity around boundaries is smoothed. With PP added in (b), dissimilarity around the boundaries are to be sharpened. With CGM added in (c), the clusters are more clearly obtained. Finally, adding SSM in (d) makes the similarity of shots within the same cluster higher, that is, more yellow than (c). The result implies that each boundary-aware pretext task encourages the shot representations to be learned favorably for the video scene segmentation. We observed similar patterns from other samples as well.

### 2. What if there are more than two scenes
 The reviewer’s comment is a very important point to discuss, because an input shot sequence often can be made up of multiple scenes as the reviewer mentioned. Our training logic assumes one single semantic boundary mainly due to the following two issues: 1) determining the exact number of semantic boundaries existing in the individual input is extremely difficult in general, and 2) judging a shot as a boundary or not is highly ambiguous, which makes this problem even more challenging. Such issues make it difficult to achieve the optimal solution that adaptively identifies every semantic boundary existing in the input sequence. As the middle ground, we formulate the problem as “finding the strongest moment with maximum semantic transition" and use the moment as a pseudo-boundary.

 Nevertheless, we argue that our formulation is quite reasonable in two ways. First, since our model is forced to find a single semantic boundary in a data-driven way, our model learns the capability of judging if the given shot is the boundary or not by referencing the context of before and after, no matter how many actual GT scene boundaries exist. Second, BaSSL observes the data in a sliding window manner, which means most of the semantic boundaries are discovered anyway by moving the window back and forth. To facilitate the reviewer’s understanding, we add a visual example in Figure 6 of the appendix. The Figure 6 (d) shows that the shot similarity matrix of the input sequence is visually divisible into “three” sub-rectangles while our BaSSL assumes only “two” scenes (one senamtic boundary) in its pre-training logic.

### 3. About concern for applicability to other method
 We agree that one may think our approach is complicated because of many components. However, in practice, the application of BaSSL to existing methods is simple. As shown in Listing 1 in Appendix D, finding a pseudo-boundary using DTW can be easily performed within 5 lines of python code. Also, our proposed pretext tasks are straightforward to implement. We demonstrate this by adapting BaSSL to ShotCoL (see M8 in Table 2).


### 4. Marginal gain by MSM
 Even though MSM has a minor effect on AP metric compared with the other tasks (PP, CGM, SSM), it effectively improves mIoU and F1 by 1.08%p and 1.72%p, respectively (see P14 and P15 in Table 3).

 1. Re '3. What if there are more than two scenes'
I have read your response. I still think the two-scene devision is not  reasonable. 
(1) As mentioned, 'As the middle ground, we formulate the problem as “finding the strongest moment with maximum semantic transition" and use the moment as a pseudo-boundary.' How to define a moment? 
(2) Also, Such issues make it difficult to achieve the optimal solution that adaptively identifies every semantic boundary existing in the input sequence. Please refer to action segmentation or action recognition[1]
[1] Finegym: A hierarchical video dataset for fine-grained action understanding, cvpr

2. Re 'Limited performance gain compared to ShotCoL'
Your response is still not convincing. ' We believe this improvement is not marginal considering that improving the score becomes extremely harder as the number gets higher. ' However, from table 1, we can see that using more training epochs would lead to performance promotion easily (56.26->57.40). This is in totally conflict with the so-called extremely harder. These statements are too subjective. 

 We truly appreciate the reviewer for the constructive comments. Below are our responses to the major concerns about our paper.

### 1. Novelty of our method
 Regarding the novelty, we first want to emphasize that our biggest contribution lies in proposing a new framework that utilizes pseudo boundaries to define effective pre-training pretext tasks for the target task (i.e., video scene segmentation). To the best of our knowledge, our work is the first to introduce boundary-aware pre-training that uses pseudo-boundaries in a completely unsupervised manner.

 We agree that one might think the individual components (i.e., pseudo-boundary discovery and pretext tasks) are not very new. However, we would like to point out that each component is complementary to each other, much synergic when combined. The performance of the model pre-trained by PP only (see P3 in Table 3)---which is the basic form of leveraging pseudo-boundaries---is far from our best performance (36.55 vs. 56.26 in AP); in addition, SSM and CGM cannot be defined without the pseudo-boundary discovery stage.


### 2. Additional analysis for better insights
 We agree that it is important to provide more evidence or demonstration for better understanding of insights such as behaviors of individual pretext tasks. Thus, we conduct the following two additional analyses with pre-trained models:

(1) NMI-based clustering capability comparison  
 The normalized mutual information (NMI) is a metric for the quality of clustering. Since clustering with good representations forms clear boundaries between different classes, NMI can be considered as a proxy to measure the quality of our pre-trained models. In brief, we collect shots from randomly sampled 100 ground-truth scenes, then perform K-Means clustering over their shot representations and measure NMI. The result is updated in Appendix F, with a detailed process and our interpretation.

 In Table 8, we compare the NMI score between different pre-trained models. The result shows that BaSSL outperforms the pre-training baselines regardless of scene length. We would like to emphasize that BaSSL shows superior performance consistently with various scene lengths, while the competing models drop their performance significantly with longer inputs. This demonstrates the effectiveness of BaSSL in maximizing intra-scene similarity.

 On the other hand, we additionally report the NMI of our method with pretext tasks added one by one in Table 9. We observe the similar tendency from our ablation experiment in Table 3; better NMI score is achieved as more pretext tasks are combined together.

(2) Visualization of similarity between consecutive shots  
We qualititatively visualize the matrix of cosine similarity between shot representations from the sequence of shots (see Figure 6 in the updated draft). The shot representations are computed by models without fine-tuning, in order to solely focus on the behavior of each objective at the pretraining stage. When the MSM is used alone, approximately 3 clusters are shown, but similarity around boundaries is smoothed. With PP added in (b), dissimilarity around the boundaries are to be sharpened. With CGM added in (c), the clusters are more clearly obtained. Finally, adding SSM in (d) makes the similarity of shots within the same cluster higher, that is, more yellow than (c). The result implies that each boundary-aware pretext task encourages the shot representations to be learned favorably for the video scene segmentation. We observed similar patterns from other samples as well.




 ### 3. What if there are more than two scene
 The reviewer’s comment is a very important point to discuss, because an input shot sequence often can be made up of multiple scenes as the reviewer mentioned. Our training logic assumes one single semantic boundary mainly due to the following two issues: 1) determining the exact number of semantic boundaries existing in the individual input is extremely difficult in general, and 2) judging a shot as a boundary or not is highly ambiguous, which makes this problem even more challenging. Such issues make it difficult to achieve the optimal solution that adaptively identifies every semantic boundary existing in the input sequence. As the middle ground, we formulate the problem as “finding the strongest moment with maximum semantic transition" and use the moment as a pseudo-boundary.

 Nevertheless, we argue that our formulation is quite reasonable in two ways. First, since our model is forced to find a single semantic boundary in a data-driven way, our model learns the capability of judging if the given shot is the boundary or not by referencing the context of before and after, no matter how many actual GT scene boundaries exist. Second, BaSSL observes the data in a sliding window manner, which means most of the semantic boundaries are discovered anyway by moving the window back and forth. To facilitate the reviewer’s understanding, we add a visual example in Figure 6 of the appendix. The Figure 6 (d) shows that the shot similarity matrix of the input sequence is visually divisible into “three” sub-rectangles while our BaSSL assumes only “two” scenes (one senamtic boundary) in its pre-training logic.


### 4. Comparison with video-level self-supervised learning (SSL) algorithm
 We agree that the methods [A,B] are competitive for learning good video-level representations from consecutive frames in trimmed videos. However, video scene segmentation data is composed of untrimmed long videos and MovieNet-SSeg dataset provides shots with sparse key-frames by default instead of consecutive RGB frames. This difference makes it difficult to extend the methods [A,B] to the video scene segmentation data. For example, the optical flow used in [A] or correspondence for [B] is hard to be computed from sparse key-frames of shots. Instead, we presented a comparison by adapting one out of state-of-the-art video-level SSL algorithms (exploiting only RGB modality) [C] to video scene segmentation data: SimCLR (instance) and Simclr (temporal). In the end, we found our BaSSL to be more effective than the adapted methods.

[A] Tengda Han et al., Self-supervised Co-training for Video Representation Learning, In NeurIPS, 2020   
[B] Allan A. Jabri et al., Space-Time Correspondence as a Contrastive Random Walk, In NeurIPS, 2020   
[C] Christoph Feichtenhofer, et al., A Large-Scale Study on Unsupervised Spatiotemporal Representation Learning. In CVPR, 2021  


### 5. Contributions for performance gain in our method
As the reviewer mentioned, fine-tuning is an important step for achieving high performance in downstream tasks as in most self-supervised learning methods including [A]. However, the main performance gain in our approach does not come from the fine-tuning only; more importantly, it is improved by our proposed boundary-aware pretext tasks. In Table 3, the models (P1-4) trained by a single pretext task significantly underperform even after the fine-tuning. Instead, exploiting more pretext tasks simultaneously during the pre-training stage brings higher performance gain. This strongly implies that utilizing all boundary-aware pretext tasks at the pre-training stage is as critical as the fine-tuning itself for the overall performance.

### 6. Limited performance gain compared to ShotCoL
The previous state-of-the-art, ShotCoL, improved 6%p in AP from its strongest competitor, LGSS. Our model improves another 4%p on top of this, as presented in Table 1 of the main paper. We believe this improvement is not marginal considering that improving the score becomes extremely harder as the number gets higher.

 With respect to the complexity, we also emphasize that our method is straightforward to apply for any existing method. We demonstrate this through incorporating our algorithm to the existing method, ShotCoL (see M8 in Table 2). Moreover, finding alignment by DTW can be achieved with 5 lines of python code (see Listing 1 in Appendix D). Once the pseudo-boundary is discovered, loss calculation for our proposed pretext tasks becomes simple math.



 We appreciate the reviewer’s insightful review. The comments helped us improve the clarity and the quality of our paper. We address the reviewer’s remarks as below.


### 1. Novelty of our method
 Regarding the novelty, we first want to emphasize that our biggest contribution lies in proposing a new framework that utilizes pseudo boundaries to define effective pre-training pretext tasks for the target task (i.e., video scene segmentation). To the best of our knowledge, our work is the first to introduce boundary-aware pre-training that uses pseudo-boundaries in a completely unsupervised manner.

 We agree that one might think the individual components (i.e., pseudo-boundary discovery and pretext tasks) are not very new. However, we would like to point out that each component is complementary to each other, much synergic when combined. The performance of the model pre-trained by PP only (see P3 in Table 3)---which is the basic form of leveraging pseudo-boundaries---is far from our best performance (36.55 vs. 56.26 in AP); in addition, SSM and CGM cannot be defined without the pseudo-boundary discovery stage.


### 2. The reason for using DTW for alignment
 The main reason we use DTW is that it has useful constraints on monotonicity for computing temporal alignment. For example, let us consider a scenario where seven consecutive shots are aligned to either first (F) or last (L) shots. Then, the monotonicity constraint in DTW results in an alignment like FFFLLLL, thus providing two consecutive sub-sequences. However, the argmax-based alignment (suggested by the reviewer), without any constraint, could produce an output alignment like FFLLFLL. Note that as our objective of alignment is to obtain two resulting sub-sequences with one pseudo-boundary, DTW is more appropriate.

 In addition, implementing DTW is extremely simple (see Listing 1 in Appendix D) and efficient through dynamic programming; indeed, we observe that the computation cost of DTW during the training is negligible---5.4ms out of 1710ms (0.32%) for a single training iteration.


### 3. Comparison with BSP
As the reviewer mentioned, the core idea of BSP seems to be similar to ours because both approaches use pseudo-boundaries. However, our work can be clearly distinguished from BSP, in the following two points: (1) we discover pseudo-boundaries fully in an unsupervised manner, while BSP requires action labels, and (2) we propose boundary-aware pretext tasks that are clearly different from the pretext task of classifying a boundary type in BSP. Specifically, BSP “synthesizes” four types of pseudo-boundaries by concatenating two clips having different semantics while our method “discovers” the moment where the semantic transition is maximal and uses it as pseudo-boundary. Moreover, BSP requires the action labels as reference for synthesizing pseudo-boundaries and defining the pretext task that classifies boundary types. Therefore, strictly speaking, BSP is not a self-supervised learning algorithm, and it cannot be simply extended to the movie scene segmentation task because no semantic labels for shots or scenes are available. In contrast, our method does not require labeled data during pre-training, thus working in a fully self-supervised manner.



### 4. Summing different metrics together
We adopt Meta-Sum metric inspired by [A,B], which use it for easy and straightforward comparison by aggregating results from multiple metrics. We include this description in the updated manuscript.

[A] Yen-Chun Chen, et al., UNITER: UNiversal Image-TExt Representation Learning, In ECCV, 2020  
[B] Linjie Li, et al., VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation, In NeurIPS, 2021  

 ### 7. Performance comparison with Grouping algorithm
As stated in Table 1 of our paper, the Grouping algorithm uses multiple modalities (i.e., RGB, Audio and Transcript). Our method achieves comparable performance with a single modality (i.e., RGB) only. Therefore, it is not straightforward to directly compare our method with the Grouping algorithm. On the other hand, incorporating more modalities to our algorithm may lead to additional performance gain since it is a known fact that using multiple modalities consistently improves the result in various video understanding tasks [D,E].

[D] Valentin Gabeur, et al., Multi-modal Transformer for Video Retrieval, ECCV, 2020  
[E] Fanyi Xiao, et al., Audiovisual SlowFast Networks for Video Recognition, arXiv preprint arXiv:2001.08740, 2020  


### 8. Results on additional datasets
As far as we know, the MovieNet-SSeg is the only large-scale video scene segmentation dataset. Unfortunately, AdCuepoints dataset suggested in ShotCoL paper is not publicly available. This is why we perform experiments mainly on MovieNet-SSeg dataset. However, we still agree with the reviewer’s opinion that testing on different datasets would be insightful. To reflect the reviewer’s suggestion, we test our method on two additional benchmarks, BBC [F] and OVSD [G]. Since the train/test splits are not available and the dataset size is extremely limited (11 and 21 videos in BBC and OVSD, respectively), we compute predictions by models trained on MovieNet-SSeg without fine-tuning on BBC and OVSD. We summarize the result in Table 7 of the updated draft. The result shows the superiority of our method compared to shot-level pre-training baselines.

[F] Lorenzo Baraldi, et al., A Deep Siamese Network for Scene Detection in Broadcast Videos, In Proceedings of the 23rd ACM international conference on Multimedia, 2015  
[G] Daniel  Rotman,  et al., Robust  and  efficient  video  scene  detection using optimal sequential grouping, In 2016 IEEE international symposium on multimedia (ISM), 2016  


 We thank all reviewers for their constructive and insightful feedback. Here, we summarize all the changes we made in the paper (highlighted in red) according to reviewers’ suggestions:

(1) Added two experiments measuring NMI to provide better insights in the Appendix F.  
(2) Added visualization of similarities between consecutive shots in the Appendix G.  
(3) Added additional experimental results on two datasets (BBC and OVSD) in the Appendix E.  
(4) Added code snippet for the alignment computation using DTW in the Appendix D.  
(5) Added a clear statement about the Meta-Sum in Section 4.1 of the main paper.  


The paper introduces a boundary-aware self-supervised learning framework for video scene segmentation. Based on the contrastive learning protocol, the authors employ DTW to generate dimidiate video segmentation from unlabeled video. Using these segments, the authors employ four loss functions to pre-train a backbone network. My most concern is about pseudo-boundary discovery. As the rest loss functions are borrowed from current self-supervised loss functions. First question is: why only use one boundary in each video? This operation is not reasonable as the video may contain multiple scenes. Also, how to define the different scenes? For figure 4 (c), we can see the shots come from the truck inside, track outside and inside again. So, the generated pseudo-scene B is confused as it contains another scene. This task is totally different from action segmentation as action segmentation has clear boundaries in each video. In my opinion, without the so-called pseudo-boundary discovery, even video-level loss for self-supervised video representation, such as [1,2] can train a strong backbone for second stage training.
[1] Self-supervised Co-training for Video Representation Learning, neurIps
[2] Space-Time Correspondence as a Contrastive Random Walk,NeurIps

My second concern is about the experimental result. The proposed pre-training the main contribution of the whole paper with complex loss functions design. However, from table 1, we can see that without finetune, the proposed method is inferior to the old Grouping method. The main performance promotion(31.55->56.26) comes from the second stage. 
Finally, from the ablation studies, the whole paper is a combinatoric paper, which combines many existing ideas instead of proposing novel ones. The ablation studies do not give much insight into the method or behavior of the proposed loss. Especially compared to existing work ShotCoL, the performance improvement is limited but the methods are much complicated. 

Some other suggestions for improvement: 1. One dataset is not enough for a top-conference work, more datasets, such as AdCuepoints should be evaluated. 2. Some typos: 1) % p.  2) . For fair comparison, all pre-training methods employ ResNet-50 (He et al., 2016) as the shot encoder fENC and we pre-train the models for 10 epochs. The quality of the whole manuscript should be further improved. The motivation of the whole task is not clear, especially compare to action segmentation task. May raise the score if author can handle all my concerns. 

This paper introduces a self-supervised learning method for the video scene segmentation task, which needs to temporally localize scene boundaries in a video. Their proposed method defines pseudo-boundary from a sequence of shots by splitting it into two sub-sequences and train a model to predict the pseudo-boundary with three novel tasks: Shot-Scene Matching (SSM), Contextual Group Matching (CGM), and Pseudo-boundary Prediction (PP). Experiments on pre-training and transfer learning show that this method is critical to improving the video scene segmentation performance, and it achieves the new state-of-the-art on the MovieNet-SSeg benchmark. (+) This paper studies an interesting problem that make a video network aware of boundary infomation for video scene segmentation. This is a general problem in many video understanding tasks such as temporal action localization, video language grounding.

(+) The proposed method is under self-supervised learning settings, which does not require any extra annotation cost, but still improve the video network for the down stream task.

(+) The discovery of pseudo boudnarys follow the intuition of the scene segmentation task. Moreover, the algorithm garantees that it can alway generate one unique pseudo boundary from a given video.

(+) According to their experimental results, the proposed method can achieve a new state-of-the-art on the MovieNet-SSeg benchmark.

(-) The core idea of this paper is similar to BSP (Xu et al., 2020), which generate psuedo boundaries and devolped pretext tasks on them. From Table 4, the performance of this work is only merginally higher (1.64%) than BSP.

(-) Not sure if or not DTW is necessary to do the temporal alignment problem. Since the second sequence has only 2 elements, the solution space is only O(K). The optima could be efficiently found by an argmanx operation after metrix multiplication. 

(-) In Table 3, the last column 'Sum' seems to be the summation. This does not make sense to me to sum different metrics together without further explaination. 
 In short, this paper studies an interesting problem, and proposes an convinient and effective method as a solution. Although the proposed solution is not very novel, and not strongly techniqual, it is marginally above the acceptance threshold for the potential impact beyond video scene segmentation. 

I also would like to see the opinion from other reviewers. 

This paper presents a boundary-aware self-supervised learning framework for the task of video scene grouping.  During the pre-train stage, the authors first apply DTW to find a pseudo boundary in a given video sequence, and then applies Shot-Scene Matching and Contextual Group Matching to optimize the intra-scene and inter-scene distribution of features. Moreover, Pseudo-boundary Prediction and Masked Shot Matching are utilized to help the model identify transitional moments.  Pros:
- This paper is clear and easy to follow.
- The overall framework looks reasonable and effective.
- The proposed method achieves good results and outperforms previous methods.

Cons
- The proposed framework is not generic and hard to be applied to other methods for the task.
- In the pseudo-boundary discovery stage, the given video sequence is separated into two scenes, which is reasonable if there the input contains only two scenes. Yet, one may wonder that how the training data is prepared and what if there are more than two scenes. Given the high performance reported in experiments, the related analysis is important which however is missing.
- From Tab.3, the effectiveness of MSM is marginal.
- The lack of insights makes this paper less interesting. E.g. the authors claim that SSM and CGM help to optimize inter-/intra- feature distribution, yet lack direct evidence/demonstration. Simply reporting some improvements in final accuracy is less convincing, as the performance can be affected by other reasons like stronger network architecture, better training protocol, etc. This paper presents an effective framework for the task and reports good performance. Yet I am still concerned about its lack of insights and generalizing ability.

Meta-review: This paper presents work on video scene segmentation.  The reviewers appreciated the introduction of a boundary-aware pre-training method.  However, concerns were raised regarding limited novelty, empirical effectiveness, and generic applicability.  The reviewers engaged in significant discussion based on the other reviews and authors' responses.  Based on these discussions the reviewers concluded that while the proposed method does have differences with respect to BSP, the overall contributions were not sufficient for inclusion in ICLR.

Decision: Reject