AutoShot: A Short Video Dataset and State-of-the-Art Shot Boundary Detection
Authors: Wentao Zhu
Xiufeng Xie
Wenxian Liu
Jincan Deng
Debing Zhang
Zhangyang Wang
Ji Liu
Venue: Submitted to ICLR 2023 (2023)
Keywords: Shot boundary detection
short video
dataset

Abstract: The short-form videos have explosive popularity and have dominated the new social media trends. Prevailing short-video platforms, e.g., TikTok, Instagram Reels, and YouTube Shorts, have changed the way we consume and create content. For video content creation and understanding, the shot boundary detection (SBD) is one of the most essential components in various scenarios. In this work, we release a new public Short video sHot bOundary deTection dataset, named SHOT, consisting of 853 complete short videos and 11,606 shot annotations, with 2,716 high quality shot boundary annotations in 200 test videos. Leveraging this new data wealth, we propose to optimize the model design for video SBD, by conducting neural architecture search in a search space encapsulating various advanced 3D ConvNets and Transformers. Our proposed approach, named AutoShot, achieves higher F1 scores than previous state-of-the-art approaches, e.g., outperforming TransNetV2 by 4.2%, when being derived and evaluated on our newly constructed SHOT dataset. Moreover, to validate the generalizability of the AutoShot architecture, we directly evaluate it on another three public datasets: ClipShots, BBC and RAI, and the F1 scores of AutoShot outperform previous state-of-the-art approaches by 1.1%, 0.9% and 1.2%, respectively. The SHOT dataset and code will be released.

Under review as a conference paper at ICLR 2023
AUTOSHOT: A SHORT VIDEO DATASET AND STATE-OF-THER ART SHOT BOUNDARY DETECTION
Anonymous authors Paper under double-blind review
ABSTRACT
The short-form videas have explosive popularity and have dominated the new social media trends. Prevailing siort-video platforms, e.g., TikTok, Instagram Reels, and YouTube Shorts, have changed the way we consume and create content. For video content creation and understanding. the shot boundary detection (SBD) is one of the most essential components in various scenarios. In this work, release new public Short video sHot bOundary de deTection dataset, named SHOT, consisting of 853 complete short videos and 11 11,606 shot annotations, with 2,716 high quality shot boundary annotations in 200 test videos. Leveraging this new data wealth, we propose to optimize the model design for video SBD, by conducting neural architecture search in search space encapsulating various advanced 3D ConvNets and Transformers. Our proposed approach, named AutoShot, achieves higher FI scores than previous state-of-the-ee approaches, e.g., outperforming Tra by 4.2%, when being derived and evaluated on our newly constructed SHOT dataset. Moreover, to validate the generalizability of the AutoShot architecture, we directly evaluate on another three public datasets: ClipShots, BBC and RAI, and the Fi scores of AutoShot outperform previous state-of-the-arr approaches by 1.1%, 0.9% and 1.2%, respectively. The SHOT dataset and code will be released.
INTRODUCTION
Short-form videos have been widely digested among the entire age groups all over the world. The percentage of short videos and video-form ads has an explosive growth in the era of 5G, due to the richer contents, better delivery and more persuasive effects of short videos than the image and text modalities (Wang et al., 202T). This strong trend leads to significant and urgent demand for temporally accurate and comprehensive video analysis in addition to a simple video classification category. Shot boundary detection is fundamental component for temporally comprehensive video analysis and can be a basic block for various tasks, e.g., scene boundary detection (Rao et al. 2020 Chen et al.) 202T), video structuring (Wang et al. (021)) and event segmentation (Shou et al. 2021) For instance, rewarded videos can be automated created of desired lengths for different platforms, leveraging the accurate shot boundary detection in the intelligent video creation.
To accelerate the development of video temporal boundary detection, several datasets have been collected with laboriously manual annotation. Conventional shot boundary detection datasets, e.g., BBC Planet Earth Documentary series (Baraldi et al., (2015)) and RAI (Baraldi et al. 2015b), only consist of documentaries or talk shows where the scenes are relatively statie, "Tang al. (2018) further contribute a large-scale video shot database, ClipShots, consisting of different types of videos collected from YouTube and Weibo covering more than 20 categories, including sports, TV shows, animals, etc. [Shou et al. (202T) construct a generic event boundary detection (GEBD) dataset, Monetics-GEBD. which defines a clip as the moment where humans naturally perceive an event. Since the video lengths of short and conventional videos differ extensively, i... 90% short videos of length less than one minute versus videos in other datasets having length of 2-60 minutes as shown in Table and Fig. Right, it dramatically leads to significant content, display, temporal dynamics and shot transition differences as shown in Fig. Left. A short video dataset is necessary to accelerate the development and proper evaluation of short video based shot boundary detection.
On the other hand, several endeavors have been made to improve the accuracy of video shot boundary detection (SBD). DeepSBD (Hassanien al. 2017) firstly applies deep spatio-temporal ConvNet

---

Under review as a conference paper at ICLR 2023
0.04
SHOT Deteset
0.03
0.01
0.00
1,00
35.0
250
300
Video Length
0.30
0:25
SHO1 Dateset
0.20
0.15
200D
0.05
0.00
0
20
20
30
40
50
shot Len (s)
Figure Left: Detecting a shot boundary can be challenging task in short videos. The shot transition can be a combination of several complicated gradual transitions (the first row) and a quick transition of the subject two shots (the second row). The visual effect of the intra-shot can vary greatly in game videos (the third row). Right: Video and shot length (s) comparison of test sets in ClipShots and our collected SHOT. There rarely has a video length range overlap between short videos in SHOT and test videos in ClipShots (up). The shot lengths of short videos are within six seconds, while the shot lengths of ClipShots can range from two seconds to 30 seconds (bottom).
to the video SBD. Deep structured model (DSM) (Tang et al. 2018 designs a cascade framework to accelerate the speed of SBD. TransNet (Lokoč al. 2019) uses dilated convolutional cells (Yu &
Koltun 2016) to process sequence of resized frames. Tra (Souček & Lokoë (020) incorpo-
rates techniques, e.g., convolution kernel factorization (Xie et al., 20188) batch normalization (loffe
Szegedy 2013), skip connection (He et al. (0166), and further improves Fl scores ClipShots (Tang
ct al. (018)) and BBC (Baraldi et al. (015a)).
In this work, we firstly collect short video dataset, named SHOT, consisting of 853 short videos with 11606 manually shot boundary fine annotations. The 200 test videos with 2,716 shot boundary annotations labeled by experts with rounds. Leveraging this data wealth, aim to improve the accuracy of video shot boundary detection, by conducting neural architecture search in search space encapsulating various advanced 3D ConvNets (Qiu et al.) 2017) and Transform- ers (Vaswani et al. (017). Single path one-shot SuperNet strategy (Guo et al. (020)) and Bayesian optimization (Shahriari et al., 2013) are employed. The searched model, named AutoShot, outper- forms Vra by 4.2% on our SHOT in terms of FI score, and by 3.5% in terms of precision metric with fixed recall rate as Tra respectively. We further evaluate the searched AutoShot architecture on ClipShots, BBC and RAI, and F1 score of AutoShot surpasses previous state-of-the-art approaches by 1.1 %, 0.9% and 1.2%, respectively. Our contributions are summarized follows:
We collect short video shot boundary detection dataset (SHOT), which consists of 853 short videos and 11,606 shot boundary annotations. The SHOT will be released and can be employed to advance the development of various short video understanding tasks.
We design a video shot boundary detection search space encapsulating various advanced 3D ConvNets and Transformers, and build neural architecture search pipeline for shot boundary detection.
The searched model, named AutoShot, proves be highly competitive shot boundary detection architecture, that significantly outperforms previous state-of-the-as approaches not only on its derived SHOT dataset, but also other public benchmarks.
2

---

Under review as a conference paper at ICLR 2023
To the best of our knowledge, the collected SHOT dataset is the first dataset for short video shot boundary detection, and AutoShot is the firstly specially designed neural architecture search method for shot boundary detection.
2 RELATED WORK
Extensive efforts have been made to collect video shot boundary detection datasets (Tang et al. 2018 Baraldi et al. 2015a b; Chakraborty et al. 2022 Jiang al., 202 Rashmi & Nagendraswamy 2021). which have significantly accelerated the development of advanced shot boundary detection methods. A specific type of video boundary detection attempts to parse the video into pieces of human actions, where instructional videos with human performing diverse actions are commonly
used (Kuehne al.] 2014 Stein & McKKnna 2013 Tang et al. 2019). Another widely studied type of video boundary detection is scene boundary detection, where videos are split into several semantically independent clips. Movies and TV episodes are commonly used for scene boundary detection, and MovieScenes (Rao et al., 2020) and AdCuepoints (Chen et al.] 2021) are two large-scale movie datasets for scene boundary detection. Furthermore, [Shou et al. (202T) collect new benchmark, Snrectics-GEBBI for generic event boundary detection. For video ads scene boundary detection, Wang et al. (202T) recently collect a multi-modal video ads understanding dataset, which involves scene boundary detection and multi-modal scene classification. However, scene boundary detection is typically based on the pre-extracted shots and the evaluation of scene segmentation is shor-level instead of frame-level.
The TV series and talk shows have also been used for shot boundary detection, where BBC Planet Earth Documentary series (Baraldi et al. 2015a) and RAI (Baraldi et al. 2015b) are two commonly used datasets consists of tens of videos having length from half an hour to one hour. ClipShots (Tang et al. 2018) enhances the conventional shot boundary detection datasets by collecting diverse videos from various media platforms, e.g., Yubeube and Weibo, and it is one of the most challenging large-scale shot boundary detection datasets. The shot transitions in short videos, are quite different from that of movies as shown in Fig. [ and Table and it is extremely necessary that short video dataset collected to advance the development short video shot boundary detection. The only short video dataset publicly available so far, to our best knowledge, is the SVD dataset (Jiang al. 2019); ; yet that is developed for a completely different task for near-duplicate video retrieval.
The accuracy of shot boundary detection has been improved, leveraging the aforementioned high quality datasets and deep learning. Before the deep learning era, PySceneDetect (Castellano (022) is popular shot boundary detection library, which relies on conventional features, e.g., changes between frames the HSV color space. Recent progresses on video boundary detection can be divided into two categories, scene boundary detection and shot boundary detection. |ao et al. (2020)
propose a local-to-global scene segmentation framework integrating multi- modal information across clip, segment, and movie. Chen et al.] (2021) further propose shot contrastive self-supervised learning to learn a shot representation that maximizes the similarity between nearby shots compared to randomly selected shots, then apply the learned shot representation for scene boundary detection.
The scene boundary detection highly depends on the accurate shot boundary detection. DeepSBD (Hassanien al. (017) predicts a likelihood of transitions in clip of 16 frames by the C3D network (Tran et al. 2015). DSM (Tang et al., (018)) utilizes a cascade framework to accelerate the speed of shot boundary detection. Gygli (Gygli 2018) constructs a fast shot bound- ary detection without any post- post-processing. TransNet (Lokoč et al. 2019) uses dilated convolution blocks YYu & Koltun| 2016) and achieves comparable accuracy as DeepSBD without post- post-processing. TransNetV2 (Souček & Lokoč 2020) surpasses previous oatte-of-the-ane approaches with advanced components, e.g., skip connection (He et al./ (0016), batch normalization (loffe Szegedy 2015).. kemel factorization (Xie et al. 2018), frame similarities as features and multiple classification heads. Leveraging the progress of 3D ConvNets (Qiu et al., 2017), Transformers (Vaswani et al. 2017) and neural architecture search (Guo et al. 2020), AutoShot automatically identifies the optimal shor boundary detection architecture from the designed search space, which achieves better accuracy than previous methods the collected SHOT, ClipShots, BBC and RAI datasets.
3

---

Under review as a conference paper at ICLR 2023
3 SHOT: SHORT VIDEO SHOT BOUNDARY DETECTION DATASET
Short video is one of the most prevailing medias in these days because of its richer contents and more vivid effects than its conventional counterparts, L.e. pure text and static picture medias. The easy and affordable access to fast mobile networks in the 5G era accelerates the widespread adoption of short video platforms, e.g., Instagram Reels, YouTube Shorts, and Tik With the large number of users and video ads in these main-stream short video platforms, it is critical to advance the current development of video temporal segmentation, especially short video shot boundary detection task, which is fundamental task for many following semantic understanding tasks.
3.1 CHALLENGES OF SHORT VIDEO SHOT BOUNDARY DETECTION
short video is typically defined video of length less than two minutes. The short video length leads to much easier spread of these videos and more popular short videos than conventional movies of hours long. On the other hand, the short video length forces the whole events to occur short time period, which causes much faster pace of events. This in turn leads to a much shorter length of a shot in the short video as shown in Fig. Right, which aggregates the difficulty of short video shot boundary detection.
The giant difference of video lengths between the test sets in the collected SHOT dataset and ClipShots (Tang et al.] 2018) is visualized in Fig. Right. Almost all the test short videos have lengths less than 100 seconds, while almost all the test videos in the ClipShots have lengths greater than 120 seconds. The short total video length directly leads to rapid shot transitions in the SHOT dataset as indicated in Fig. Right. Most shot lengths are within five seconds in the test set of SHOT dataset, and the shot lengths of test set in the ClipShots The conventional shot boundary detection dataset may be inappropriate for the development of short video shot boundary detection because of the great difference of video and shot length distributions.
range from two seconds to 30 seconds.
Short video shot boundary detection is much more challenging and difficult as illustrated in Fig. Left and Fig. [ because the scene of the short video is much more complicated than conventional videos. For instance, the shot transition commonly utilizes a combination of several complicated shot gradual transitions for the persuasive effect in the short video (the first row). The second common challenge is for vertically ternary structured videos (the second row of Fig. 2. where only the middle part of the video changes. The uppermost and lowermost regions display the download link or brand in the video ads. The vertically ternary structured video increases shot boundary detection difficulty greatly due to the relatively small region change in the squeezed content region. The exaggerated expression in the virtual scene causes false alarms in the game video shot boundary detection (the third row). Actually, the game video takes a large proportion of video ads and short videos. Therefore, collecting short video dataset is nontrivial for the challenging short video shot boundary detection.
3.2 SHOT DATASET
To accelerate the study of short video related shot boundary detection, collect 853 short videos from one of the most widely used short video platforms. The dataset property comparison is listed in Table I The total number of frames is 960,794, close one million frames. The frame-wise shot boundary annotation is a heavy task. The data annotation and quality control strategy are in appendix.
The cases from the SHOT dataset can be found in Fig. Left, Fig, 2 and Fig. 4 The annotation records the start and end frame numbers of each shot, which is visualized by the pink and light/white colors in Fig. 4 All the training and testing short videos, shot boundary annotations, the evaluation metric scripts and the explicit video-level data split will be publicly available.
AUTOMATED SHOT BOUNDARY DETECTION
4.1 SHOT BOUNDARY DETECTION SEARCH SPACE DESIGN
We design a SuperNet based on of the previous state-of-the-art approaches, TransNetV2 (Souček) & Lokoë 2020). where the feature representational learning network can be considered as a sequence of six factorized dilated deep 3D convolutional neural network (DDCNNV2) blocks. Leveraging the
4

---

Under review as a conference paper at ICLR 2023
Dataset
BBC
RAI
Clip.
SHOT
Complex
x
X
X
Grad. Trans.
Virt. Scene
X
X
X
Tern. Video
X
X
V
Avg. Video
2945
591
237
39.5
Len. (s)
Avg. Shot
6.57
5.65
15.34
2.59
Len. (8)
Table I: Comparison of different short boundary detection datasets, i.e., BBC, RAT, ClipShots and SHOT, w.rtt complex gradual transition, virtual scene, ternary video, average video length and av- erage shot length.
Figure 2: Unique challenges of short video shot boundary detection in SHOT, e.g., a combina- tion of complicated shot gradual transitions, verti- cally ternary structured video, and great intra-shot change in virtual scenes of game video.
(a) DDCNNV2
(b) DDCNNV2A
(c) DDCNNV2B
(d) DDCNNV2C
Figure 3: Illustration of the search blocks for the first six blocks in the AutoShot. It consists of four types of factorized 3D convolution blocks, (a) DDCNNV2 with 2D spatial convolutions followed by ID temporal convolutions, (b) DDCNNV2A with a shared 2D spatial convolution and ID temporal convolutions, (c) DDCNNV2B accumulating a 2D spatial convolution and ID temporal convolutions, and (d) DDCNNV2C compromising between DDCNNV2A and DDCNNV2B.
advanced 3D ConvNets (Qiu et al. 2017) and Transformers (Vaswani et al. 2017), we deign shot boundary detection neural architecture search space. Specifically, AutoShot conducts architecture search seven blocks, where add self-attention layer number search after the sixth block. The search blocks in the first six blocks are illustrated in Fig, 3 We firstly design four types of factorized 3D convolutions in the search space. Let be the input for the current block, the four kinds of search blocks can be formulated as following:
(1) DDCNNV2: We conduct the search on the number of dilated convolution branches na, i.e., and 5, and the channel number of 2D spatial convolution ne, i.e., 1, 2 and 3 times of the input channel numbers, for the original DDCNNV2 block in the TransNetV2 illustrated in Fig. 3a The DDCNNV2 can be formulated as
h = ReLUUBN((Coneee [h, h2. h D)). h, = S.) = (x),, i 1= 1, 1, nd. (1)
where h is the output of the current block, S. is the 2D spatial convolution with the channel number 1. Tç is the ID temporal convolution with the channel number [ [FF] and dilation rate 2 and F is a pre-detined channel number Fig. Bal The key components in DDCNNV2 are dilated temporal 1D convolutions, and factorized 3D convolutions with spatial 2D convolutions and temporal ID convolutions. The design enforces diverse contextual temporal feature extraction and reduces the number of learnable parameters, which might reduce the over-fitting.
(2) DDCNNV2A: To unify the feature extractor of the spatial 2D convolutions, we can employ a shared 2D convolution instead of multiple branches of spatial 2D convolutions, illustrated in
5

---

Under review as a conference paper at ICLR 2023
Fig. 3b The shared spatial 2D convolution aims to extract unified spatial feature for the following diverse temporal feature extractions. The DDCNNV2A can be expressed as
h RcLU [b,, he. ..n,)))), h, (T,
(S(x))), i= 1, rdd
where S is a shared 2D spatial convolution with searched channel number fe.
(3) DDCNNV2B: Inspired by the design of Pseudo-3D network (Qiu et al. 2017), construct another two search blocks to learn various spatio-temporal representations as illustrated in Fig. 3c and 3d
The DDCNNV2B can be given by
h = ReI.U(BN( + TT) x)) ReL.U(BN(S()) + T(x))), (x)])-
T(x)= Concat((T, (x), T2(x),
To ensure the channel numbers of spatial features and temporal features are equal, the channel number of 2D spatial convolution fixed be four times of the current block's input dimension number 4/'
(4) DDCNNV2C: Different from DDCNNV2B, the temporal convolution of DDCNNV2C can still utilize the feature of spatial convolution as illustrated in Fig. 3d which can be formulated as
h = (cLU(BN((S + T x)) (cLU(BN(S(x) + T(S(x)))), (x)(x)) Concat( (T1 (S(x)), T2(S(x)). Tn,, (S(x))).
We further construct ID temporal Transformer block after six factorized convolution layers to enhance the temporal modeling, whose input is flattened frame-wise convolutional feature. We conduct the number of self-attention layers search {0, 1, 2, 3, 4} in the Transformer block.
In summary, AutoShot has seven search blocks. In the first six search blocks, it conducts the channel number search and branch/dilation number search for DDCNNV2 and DDCNNV2A. Limited by the dimension number consistency of element-wise addition, DDCNNV2B and DDCNNV2C searches
the branch number. Consequently, we have 3 x 2 x 2 + 2 x 2 = 16 options for each search block in the first six search blocks. The search space of AutoShot totally has (16*) x 5 8.39 x 107 candidate architectures.
4.2 AUTOSHOT TRAINING AND SEARCH
After the construction the base network for representational learning, we concatenate represen- tations from the base network with RGB histogram similarity of raw input frame and learnable
cosine similarity of concatenated block features (Chasanis et al. 2009 Souček & Lokoë 2020) to
construct 4, 4,864 dimension feature vector. Then fully connected layer 1,024 neurons with ReLU
activation Nair & Hinton 2010) is added. Next a dropout layer (Krizhevsky et al. (012)) with dropout rate 0.5 used before the final two frame-wise classification heads for single middle frame
of transition y and all transition
The SuperNet training of AutoShot utilizes an efficient weight sharing strategy. The weight sharing strategy (Guo et al.) 2020: Liu et al., 2018) encodes the search space in a SuperNet, and all the candidate architectures share the weights of the SuperNet. One shot NAS (Guo et al. (020) decouples the SuperNet training and architecture search, which yields better accuracy. We also conduct two sequential steps for SuperNet training and architecture search. We utilize a single path and uniform sampling strategy reduce the co-adaptation between node weights (Guo et al. 22020).
We implement a Bayesian optimization (Shahriari et al. 2015) based architecture search for AutoShot, Bayesian optimization iterates between fitting probabilisti surrogate models and determining which configuration to evaluate next by maximizing an acquisition function. Gaussian process with a Hamming kernel is utilized the surrogate function. We employ random exploration in the initialization to obtain good Gaussian process model. For the acquisition function, we use probability of feasibility (Gardner al. 2014).
For the candidate architecture retraining, we firstly employ the same two classification, i.e., single middle frame y transition and all transition 2, cross-entropy losses the SuperNet training
Cretrain
Aayiy log Hh.s + A2 log {(J]
(5)
6

---

Under review as a conference paper at ICLR 2023
where N is the batch size in the stochastic gradient descent (SGD), Np is the pre-defined number of frames for each training sample which is processed in each batch of training, A1 and Xg are trade-offs between two classification heads, and Ji, and &i.j are two frame-wise predictions of single middle frame of a transition and all transition, respectively.
After retraining with the plain multi-head cross-entropy classification loss in Eq. equation further enhance the candidate networks by employing the best performing candidate network as a teacher network in knowledge distillation (Hinton et al. 2014), and utilize weight grafting (Meng et al. 2020) to further improve the best accuracy. The knowledge distillation is used to align candidate networks with a desired accuracy, and the weight grafting adaptively balances the grafted information among aligned networks, which improves the representation capability and boosts the accuracy. The entropy-bane -baseed weight grafting can be formulated as
L(W) = - -E (Ai log Di.j + A2 log Eg
a= Ax (arctan(c x (H(W; H(W 0.5, W.Ms aW, + (1 w)),
where ji. and žG, are two Trame-wise predictions of single middle frame of a transition and all transition from the teacher network, A and e are fixed hyperparameters, is the coefficient based on entropy to balance networks, (-) is the entropy based on the bins of network weights W.M, or W-Me, 1 is the layer in the network, Mj and Ma are two networks where network M2 accepts the information from network M1.
5 EXPERIMENTS
We conduct neural architecture search based on the evaluation metrics on the collected SHOT and validate the effectiveness of the searched optimal architecture on SHOT, ClipShots (Tang et al. 2018 BBC (Baraldi et al. 2015a), and RAI (Baraldi et al. 2015b). In both SuperNet training and candidate network retraining. we construct each training sample by concatenating two shots randomly and the number of frames NF in each training sample is set as 60. The hyperparameters Åo A2. A, the number of bins in the entropy calculation and the number of grafting networks in Eq. equation 6 are set as 5, 1. 0.4, 1.0, 10 and 3, respectively. which are generally followed the setting of previous works (Souček Lokoë 2020 Meng al. 2020). We use stochastic gradient descent with learning rate 0. momentum 0.9, batch size 16 and the number of epochs 12. In the search, the number of populations per epoch is 48 and the total number of epochs is 100 with 20 epochs for the initialization. We utilize the OpenBox library (Li et al.) (021) to implement the Bayesian optimization in the search. We use one 32 GB NVIDIA Tesla V GPU for both SuperNet and candidate network training, and use eight 32 GB NVIDIA Tesla V100 GPUs to accelerate the search speed. All the code, data and trained models will be released for the reproducibility.
Neural architectrre search SHOT We train the SuperNet on the combined training set of SHOT and ClipShots, and conduct the search based on the evaluation metrics on the collected SHOT dataset. For fair comparison, we closely follow the dataset protocol and metric calculation of previous work (Souček & Lokoë 2020 Baraldi et al. 20155). Additionally, we also conduct a search based on the precision metric given a fixed recall rate 0.71 same Vr2 because a higher Fl metric cannot guarantee higher precision and recall scores simultaneously in practice. From Table Left, AutoShot outperforms Tra by 4.2% and 3.5% based on Fl and precision metries. Note that the F1 score of PySceneDetect (Castellano 2022) on SHOT is less than 0.6. which is far behind AutoShot, and it can hardly handle challenging gradual transitions. For AutoShot, we find that 54% of missed shots are gradual transitions, and gradual transitions take 30% of shots in SHOT. Gradual transition has no huge inter-frame difference, which difficult be fully detected. In the following, only compare AutoShot based on the Fl metric with other methods for consistency.
To compare the predictions visually, we employ video thumbnail images to clearly demonstrate the difference. The ground truth boundary is shown in the pink or light/white color as shown in Fig. 4 The detected boundary of AutoShot is marked as pink, cyan or light, and the detected boundary of TransNetV2 is visualized cyan or light. AutoShot successfully detects minor shot transitions as shown in the pink color, where the TransNetV2 fails. For the clear transitions, both the TransNetV2 and AutoShot succeed, shown in the light color. The false positives of both TransNetV2 and
7

---

Under review as a conference paper at ICLR 2023
39
78
779
780
781
82
800
801
802
803
804
805
806
807
8
825
826
828
829
830
831
832
Figure 4: Visual comparison of shot boundaries of ground truth (pink, light/white), TransNetV2 (cyan, light/white), and AutoShot (pink, cyan, light/white) on four clips from SHOT dataset. AutoShot detects minor transition, shown in the pink color. The light/white color denotes that both the T2a and AutoShot successfully detects the shot boundary. The cyan denotes the false positives of both TransNetV2 and AutoShot, which might require adaptively understanding of contextual semantics across the video.
AutoShot are shown in cyan, which are hard negative shots. Reducing the false positives might require adaptively understanding of contextual semanties across the whole video, and some false positives are ambiguous even for human annotators.
Generalization on other datasets After obtaining the optimal network architecture from the SHOT dataset, we validate the network generalizability on other three publicly and widely used datasets, ClipShots Tang et al. 2018). BBC (Baraldi et al. 2015a),, and RAI (Baraldi et al. 2015b). The three existing datasets are quite different from our SHOT dataset, as shown in the section BBC Planet Earth Documentary series (Baraldi et al. 2015a consists 11 episodes from the BBC educational TV series Planet Earth. Each episode is approximately 50 minutes long, and the whole dataset contains around 4900 shots and 670 scenes. RAI (Baraldi et al. 2015b) dataset based on a collection of ten randomly selected broadcasting videos from the Rai Scuola video archive, where the length of each video is around half an hour. The ClipShots collects thousands of online conventional videos, not short videos, from YouTube, which is much more challenging dataset than BBC and RAI. We use the same dataset split and protocol as previous work (Součck & Lokoč 2020 Baraldi et al. 2015b).
From Table 2 Right, simply applying the searched AutoShot architecture the three datasets obtains better Fl scores than previous satte-of-the-art approaches consistently, which sufficiently validates the effectiveness and good generalizability of AutoShot. Specially, AutoShot outperforms previous state-of-the-art approaches by 1.1 1.1%, 0.9% and 1.2% on ClipShots, BBC and RAL Note that we reproduce T222 based on PyTorch and obtain F1 score of 0.776 on ClipShots, while the original paper (Souček & Lokoč, 2020) reports 0.779 based TensorFlow.
Effect of search space We investigate the effect of three different search spaces, i.e., AutoShot-S, -M and -L, in Table 3 (Left). AutoShot-S only employs DDCNNV2A components in the search space, which has six search options per block. AutoShot-M - employs DDCNNV2 and DDCNNV2A components in the search space, which has 12 search options per block. The AutoShot-L denotes the search space defined in section 4.1 which achieves the best Fl score. The combined various 3D ConvNet variants, iee., DDCNNV2, DDCNNV2A, B and C. in each search block improves F1
8

---

Under review as a conference paper at ICLR 2023
Dataset
ClipShots
BBC
RAI
Method
TransNetV2
AutoShot @F1
AutoShot
DSMs
0.761
0.893
0.928
@ Precision ST ConvNets
0.759
0.926
0.939
FI
0.799
0.841
0.826 0.939
TransNet
0.735
0.929
0.943
Prec.
0.904
0.923
TransNerV2
0.776
0.962
0.939
AutoShot
0.787
0.971
0.955
Table 2: Left: AutoShot surpasses TransNetV2 by 4.2% and 3.5% based on the score and precision metric with a fixed recall as Tra Right: the searched optimal architecture is validated on three widely used shot boundary detection datasets. AutoShot consistently achieves the best F1 compared to DSMs (Tang et al. 2018). 3) ST ConvNets (Hassanien et al. (0177) TransNet (Lokoč et al. 2019) and TransNet V2
(Soucek Lokoč 2020
AutoShot-
S
M
L
w/ KD+ weight graft
Method
w/o KD
w/ KD
w/o retrain
0.816
0.822
0.831
w/ retrain
0.833
0.837
0.841
F1
0.825-08377
0.832-0.838
0.841
Table Fl scores of different search spaces (Left), knowledge distillation (KD) and weight grafting (Right) on the collected SHOT dataset.
score in both retraining and candidate architectures after search, since the optimal blocks for different layers vary and more search options in AutoShot permit identify the optimal composition.
Searched architectures Specifically, the obtained architecture based on F1 score, AutoShot@Fi DDCNNV2((m.=41.. n==4), A(r,=4/', n2==5), A(n,=4F, n₂=5),, A(n,=4F, m===5), (n.=12F, m.=5), =8F n==5)), which has floating-point operations (FLOPs) of 37 GMACs. The obtained architecture based on precision metric, AutoShot Prec., is DDCNNV2 (m. n2= (n.=8F, n.==), B(n==4), C(na=4), B(n==5), B(n==4)}, which has FLOPs of 30 GMACs. We find that the optimal architectures indeed employ diverse blocks and less number of operations than TransNetV2 of 41 GMACs. The search SHOT chooses more dilated convolutional branches and diverse blocks, because more dilated convolutional branches na and diverse blocks enhances the representational learning power for various temporal granular shot transitions, which vastly exist in short videos. This probably another reason that AutoShot on SHOT achieves much bigger improvement than that on the conventional video datasets. Although the two optimal architectures the training of SuperNet and architectures with close Fl or precision scores utilize the self-attention.
self-attention layer,
Effect of knowledge distillation and weight grafting We ablate knowledge distillation and weight grafting in Table 3 (Right) based on the constructed search space section 4.1 Without knowledge distillation in Eq. equation 5 the range of F1 scores after retraining the best performing, i.e., FI score of 0.837, architecture as the teacher network, and knowledge distillation aligns the candidate architectures with Fl score of range 0.832-0.838. Then, the weight grafting in Eq. equation 6 further improves the best FI score by 0.3%.
be 0.825-0.837. We use
6 CONCLUSION
In this work, we collect a new short video shot boundary detection dataset, named SHOT, which is quite different scenario from conventional long video based shot boundary detection. The SHOT can sigmificantly accelerate the development and evaluation of various short video based applications, e.g., intelligent creation, and video scene segmentation and understanding. Leveraging this new asset, we propose to optimize the model design for the task of video shot boundary detection, by conducting neural architecture search search space encapsulating various advanced 3D ConvNets and Transformers. AutoShot surpasses previous best shot boundary detection method by 4.2% and 3.5% based on the and precision scores, respectively. We further validate the generalizability of the searched optimal architecture ClipShots, BBC and RAL. Experimental results demonstrate that, AutoShot has a good generalizability and outperforms previous saate-of-the-a approaches on the three existing public datasets.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. A deep siamese network for scene detection in broadcast videos. In Proceedings of the 23rd ACM International Conference on Multimedia, pp- 1199-1202, 2015a.
Lorenzo Baraldi, Costantino Grana, and Rita Cucchiara. Shot and scene detection via hierarchical clustering for re-using broadcast video. In Proceedings of International Conference on Computer Analysis of Images and Patterns, pp. 801-811. Springer, 2015b.
Brandon Castellano.
Pyscenedetect.
htt:://ww.bastteL..commm/e
PySceneDetect] 2022.
Saptarshi Chakraborty. Alok Singh, and Dalton Meitei Thounaojam. novel bifold-stage shot boundary detection algorithm: invariant to motion and illumination. The Visual Computer, 38(2): 445 456, 2022.
Vasileios Chasanis, Aristidis Likas, and Nikolaos Galatsanos. Simultaneous detection of abrupt cuts and dissolves in videos using support vector machines. Pattern Recognition Letters, 30(1):55-65, 2009.
Shixing Chen, Xiaohan Nie, David Fan, Dongqing Zhang, Vimal Bhat, and Raffay Hamid. Shot contrastive self-supervised learning for scene boundary detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9796-9805, 2021.
Jacob Gardner, Matt Kusner, Zhixiang Eddie Xu, Kilian Q Weinberger, and John P Cunningham. Bayesian optimization with inequality constraints. In Proceedings of International Conference on Machine Learning, volume 2014, pp. 937-945, 2014.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Proceedings of European Conference on Computer Vision, pp. 544-660. Springer, 2020.
Michael Gygli. Ridiculousty fast shot boundary detection with fully convolutional neural networks. In Proceedings of 2018 International Conference on Content- -assed Multimedia Indexing (CBMI), pp. 1-4. IEEE, 2018
Ahmed Hassanien, Mohamed Elgharib, Ahmed Selim, Sung-Ho Bae, Mohamed Hefeeda, and Wojciech Matusik. Large-scale, fast and accurate shot boundary detection through spatio-temporal convolutional neural networks. arXiv preprint arXiv:1705.0328. 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. In Proceedings of 28th International Conference on Neural Information Processing Systems Deep Learning Workshop. 2014.
Sergey loffe and Christian Szegedy. Batch normalization: Accelcrating deep network training by reducing internal covariate shift. In Proceedings of International Conference Machine Learning, pp.
PMLR, 2015.
Qing- Yuan Jiang, Yi He, Gen Li, Jian Lin, Lei Li, and Wu-Jun Li. Svd: A large-scale short video dataset for near-duplicate video retrieval. Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 5281-5289, 2019.
Xuckun Jiang, Libiao Jin, Anyi Rao, Linning Xu, and Dahua Lin. Jointly learning the attributes and composition of shots for boundary detection in vidcos. IEEE Transactions on Multimedia, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Proceedings of Advances in Neural Information Processing Systems, volume 25, pp. 1097-1105, 2012.
10

---

Under review as a conference paper at ICLR 2023
Hilde Kuchne, Ali Arslan, and Thomas Serre. The language of actions: Recovering the syntax and semantics of goal-directed human activities. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 780-787, 2014.
Yang Li, Yu Shen, Wentao Zhang, Yuanwei Chen, Huaijun Jiang, Mingchao Liu, Jiawei Jiang, Jinyang Gao, Wentao Wu, Zhi Yang, et al. Openbox: A generalized black-box optimization service. In Proceedings of 27th SIGKDD Conference on Knowledge Discovery and Data Mining (KDD-2021), 2021.
Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. In Proceedings of International Conference Learning Representations, 2018.
Jakub Lokoë, Gregor Kovalčik, Tomáš Souček, Jaroslav Moravec, and Piemysl Čech. A framework for effective known-item search in video. In Proceedings of the 27th ACM International Conference on Multimedia, pp. 1777-1785, 2019.
Fanxu Meng, Hao Cheng, Ke Li, Zhixin Xu, Rongrong Ji, Xing Sun, and Guangming Lu. Filter grafting for deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6599-6607, 2020.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of International Conference on Machine Learning, 2010.
Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1543,,541, 2017.
Anyi Rao, Linning Xu, Yu Xiong. Guodong Xu, Qingqiu Huang, Bolei Zhou, and Dahua Lin. A local- to-global approach to multi-modal movie scene segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10146-10155, 2020.
BS Rashmi and HS Nagendraswamy. Video shot boundary detection using block based cumulative approach. Multimedia Tools and Applications, 80(1):641-664 2021.
Bobak Shahriari. Kevin Swersky, Ziyu Wang, Ryan P Adams, and Nando De Freitas. Taking the human out of the loop: A review of bayesian optimization. Proceedings of the IEEE, 104(1): 148-175, 2015.
Mike Zheng Shou, Stan W Lei, Weiyao Wang, Deepti Ghadiyaram, and Matt Feiszli. Generie event boundary detection: A benchmark for event segmentation. In Proceedings of the IEEF/CVF International Conference Computer Vision, 2021.
Tomáš Souček and Jakub Lokoč. Transnet v2: An effective deep network architecture for fast shot transition detection. arXiv preprini arXiv:2008.0444 04838, 2020.
Sebastian Stein and Stephen McKenna. Combining embedded accelerometers with computer vision for recognizing food preparation activities. In Proceedings of the 2013 ACM International Joint Conference on Pervasive and Ubiquitous Computing, pp. 729 738, 2013.
Shitao Tang, Litong Feng, Zhanghui Kuang, Yimin Chen, and Wei Zhang. Fast video shot transition localization with deep structured models. In Proceedings of Asian Conference on Computer Vision, pp. 577-592. Springer, 2018
Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. Coin: large-scale dataset for comprehensive instructional video analysis. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1207-1216, 2019.
Du Tran, Lubomir Bourdev, Rob Fergus. Lorenzo Torresani. and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE International Conference Computer Vision, pp. 4489-4497, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.

---

Under review as a conference paper at ICLR 2023
Zhenzhi Wang. Zhimin Li, Liyu Wu, Jiangfeng Xiong, and Qinglin Lu. Overview of tencent multi- modal ads video understanding. Proceedings of the 29th ACM International Conference on Multimedia, pp. 1725-4729, 2021.
Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spatiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV). pp. 305-321, 2018.
Fisher Yu and Vladlen Koltun, Multi-scale context aggregation by dilated convolutions. In Proceed- ings of International Conference on Learning Representations, 2016.
12


Reviews:
[{"review_id": "nZL5c6I_3vT", "reviewer": "Reviewer_6hJo", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a new dataset, SHOT, for short video shot boundary detection (SBD). Together with the dataset, the authors propose a baseline model, AutoShot, through NAS among 3D ConvNets and Transformers components. The authors have demonstrated the unique statistics of the SHOT dataset, arguing that SHOT can be used to solve challenges of short video SBD. The AutoShot model is evaluated on three public benchmarks against previous state-of-the-art approaches, and the performance is promising.", "summary_of_the_review": "Overall this is a good paper with a hopefully useful dataset that the video community can benefit from, especially due to the overwhelming trend of short and vertical videos. Although obtained by NAS on top of existing modules, the AutoShot model sets new standard of performance on multiple public benchmarks. ", "strength_and_weaknesses": "Strengths:\n- The dataset fills the blank for shot boundary detection on short videos. Previous datasets do not contain the vertically viewed and short videos which is very popular these days.\n- Apparently the AutoShot model achieves state-of-the-art performance on all three benchmarks, ClipShots, BBC and RAI.\n\nWeaknesses:\n- Size of the SHOT dataset seems small, as it contains 853 videos and most of which are shorter than 1 minute. A few reference dataset:\n  - ClipShots, 4039 videos with 128636 cut transitions and 38120 gradual transitions. Length of most videos is greater than 2 minutes.\n  - Short Video Dataset (SVD, Jiang et al. 2019), over 500,000 short videos and 30,000 labeled pairs of near-duplicate videos. SVD is designed for the video retrieval task.\n- In terms of the proposed model, there is no design that is specifically motivated by the unique challenges that short video pose. Instead, NAS is performed on top of existing state-of-the-art work. In other words, the model design is not well motivated but more through brute-force engineering upon existing models.\n- It's unclear if the challenges mentioned in Figure 1 have been well tackled by the AutoShot model. How well does the best performing model do on the challenging cases described in Figure 1? Are there some qualitative demos on each type of challenge?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "WLQ-_wQi01", "reviewer": "Reviewer_3zbz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies shot boundary detection, a long-standing problem, in a particular case: short-form videos. The challenges come from complicated shot gradual transitions, vertically ternary structured videos, and intra-shot change in virtual scenes. The authors collect a new public Short video sHot bOundary deTection dataset, named SHOT, consisting of 853 complete short videos and 11,606 shot annotations. Based on this dataset, the paper proposes to optimize the model design for video SBD, by conducting a neural architecture search in a search space encapsulating various advanced 3D ConvNets and Transformers. It achieves higher F1 scores than previous state-of-the-art approaches.", "summary_of_the_review": "Considering the novelty and quality of this paper, I think it does not reach the bar of the top-tier ICLR conference and vote for rejection.", "strength_and_weaknesses": "Strength: \n- The paper shows its approach in detail, and the structure is formal. \n- The experiments are tested on different datasets.\n\nWeaknesses: But I have the following significant concerns on quality, and novelty:\n\n- Dataset construction needs several steps, including data collection, annotation, and verification. The authors may need to clarify that the collection process is not challenging. The data may come from web crawling or database accessing if they are from a video company. And at the same time, the copyright is not protected. How to handle and protect the copyright of video creators is questionable.\n\n- The annotation is challenging since it requires frame-wise, but it is not reasonable. The annotation approach that uses a thumbnail cannot provide enough information for the annotators to well handle the challenging cases. Instead, the results may be similar to conventional methods that detect shots based on appearance. The details of the annotation are not clear in the supplementary. How to ensure quality? With double check or voting?\n\n- The method lacks novelty that does not show much difference from the exiting NAS papers on classification. The challenges mentioned by authors are actually very good points for design, e.g., incorporating an adaptive sliding window to handle the gradual transition, or separately handling the background and subject for intra-shot changes, or using rules of thumb for ternary videos. A well-designed lightweight mixture-of-expert network may well handle the challenges. \n\n- The experiments are less interesting that cannot provide insights on the challenges mentioned by the authors, noting that the challenges are the motivation of the authors to bring out a new setting. And the qualitative results are misleading, and it is hard to tell the difference of different methods. And this also adds to my concerns about the quality of annotation.\n\nMinor:\n- What is the dataset scale comparison with the existing datasets?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "NBWwJG94a1", "reviewer": "Reviewer_Ar2e", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is about shot boundary detection of short-form videos (cf. TikTok or YouTube Shorts).\nAccording to the authors, short videos are one of the most prevailing medias in these days and shot boundary detection is a fundamental task for many following semantic understanding tasks.\nMore specifically, the paper carefully motivates the need for an additional dataset for this task, and announces the new dataset \"SHOT\" along with a baseline (AutoShot).\nThe baseline is the outcome of the neural architecture search over a search space spanned by existing architectures (3D ConvNets and Transformers), specialised for shot boundary detection.\nThe such searched model gives a gain of 4% on the new dataset and 1% on other public benchmarks.\n\nThe main contributions of the paper are the new benchmark with a highly competitive baseline (accuracy-wise and compute-wise) found by a neural architecture search.\n", "summary_of_the_review": "On one hand, the recent popularity of short videos comes with the need for new public benchmarks to facilitate development and comparison of models, and the paper fills in on this. \nOn the other hand, the contribution in terms of modelling, algorithms or analysis is limited.\nWhich makes my recommendation \"borderline\".\n", "strength_and_weaknesses": "Strengths:\n- Provides a dataset/benchmark for shot boundary detection on short videos. \n- Solid baseline with a model architecture that is specialised for shot boundary detection.\n\nQuestions:\n- Boundary detection is crucial for long videos. Is this still true for short videos or would a less pipelined/more end-to-end approach be more appropriate, the more as \"the frame-wise shot boundary annotation is a heavy task\"?\n- Is this task relevant for a broader/broad enough audience?\n\nOrganisation of paper: \n- Section 3.2: It would be useful if this would be the place with all data-related statistics, including frame size, FPS, type/format of annotation, etc. In the current version, the information is spread across the abstract, introduction, appendix. \n- Section 4: I suggest to move the details on the basic shot boundary detection model from Section 4.2 into a separate paragraph (e.g. 4.0).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7mEqRHGnrE", "reviewer": "Reviewer_wabR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For the task of shot boundary detection, this paper releases a new public dataset SHOT, which is complementary to existing related datasets. In addition, the model design is optimized by conducting neural architecture search in a search space encapsulating various advanced 3D ConvNets and Transformers. Experiments show the effectiveness of the proposed method.", "summary_of_the_review": "This paper releases a new public dataset, as well as proposes an interesting method for shot boundary detection, which achieves SOTA. It is suggested to accept this paper.", "strength_and_weaknesses": "Strength:\n1)\tThe newly released dataset SHOT is valuable, which will facilitate the development of community of short video understanding.\n2)\tThe proposed neural architecture search based method is new for shot boundary detection, which outperforms previous SOTA approaches.\n\nWeakness:\n1)\tIt is difficult to understand for the readers without the neural architecture search background. It's technical contribution is limited. \n2)\tThe hyper-parameter sensitivity analysis is missing in the experiments.\n3)\tThe dataset is relatively small, which consists of 853 videos. A larger dataset will be more meaningful. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject