Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper is about shot boundary detection of short-form videos (cf. TikTok or YouTube Shorts).
According to the authors, short videos are one of the most prevailing medias in these days and shot boundary detection is a fundamental task for many following semantic understanding tasks.
More specifically, the paper carefully motivates the need for an additional dataset for this task, and announces the new dataset "SHOT" along with a baseline (AutoShot).
The baseline is the outcome of the neural architecture search over a search space spanned by existing architectures (3D ConvNets and Transformers), specialised for shot boundary detection.
The such searched model gives a gain of 4% on the new dataset and 1% on other public benchmarks.

The main contributions of the paper are the new benchmark with a highly competitive baseline (accuracy-wise and compute-wise) found by a neural architecture search.

strength_and_weaknesses: Strengths:
- Provides a dataset/benchmark for shot boundary detection on short videos. 
- Solid baseline with a model architecture that is specialised for shot boundary detection.

Questions:
- Boundary detection is crucial for long videos. Is this still true for short videos or would a less pipelined/more end-to-end approach be more appropriate, the more as "the frame-wise shot boundary annotation is a heavy task"?
- Is this task relevant for a broader/broad enough audience?

Organisation of paper: 
- Section 3.2: It would be useful if this would be the place with all data-related statistics, including frame size, FPS, type/format of annotation, etc. In the current version, the information is spread across the abstract, introduction, appendix. 
- Section 4: I suggest to move the details on the basic shot boundary detection model from Section 4.2 into a separate paragraph (e.g. 4.0).
clarity,_quality,_novelty_and_reproducibility: Overall, the paper is easy to follow and clearly written (see above for some suggestions for improving the organisation of the paper). 
The experimental part looks solid.
The main novelty of the paper is the new benchmark.
Given the fairly involved multi-step pipeline, I wouldn't be confident in being able to reproduce the results from the paper. But the authors state that they will release both the dataset and source code so reproducibility shouldn't be an issue.
summary_of_the_review: On one hand, the recent popularity of short videos comes with the need for new public benchmarks to facilitate development and comparison of models, and the paper fills in on this. 
On the other hand, the contribution in terms of modelling, algorithms or analysis is limited.
Which makes my recommendation "borderline".
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a new dataset, SHOT, for short video shot boundary detection (SBD). Together with the dataset, the authors propose a baseline model, AutoShot, through NAS among 3D ConvNets and Transformers components. The authors have demonstrated the unique statistics of the SHOT dataset, arguing that SHOT can be used to solve challenges of short video SBD. The AutoShot model is evaluated on three public benchmarks against previous state-of-the-art approaches, and the performance is promising.
strength_and_weaknesses: Strengths:
- The dataset fills the blank for shot boundary detection on short videos. Previous datasets do not contain the vertically viewed and short videos which is very popular these days.
- Apparently the AutoShot model achieves state-of-the-art performance on all three benchmarks, ClipShots, BBC and RAI.

Weaknesses:
- Size of the SHOT dataset seems small, as it contains 853 videos and most of which are shorter than 1 minute. A few reference dataset:
  - ClipShots, 4039 videos with 128636 cut transitions and 38120 gradual transitions. Length of most videos is greater than 2 minutes.
  - Short Video Dataset (SVD, Jiang et al. 2019), over 500,000 short videos and 30,000 labeled pairs of near-duplicate videos. SVD is designed for the video retrieval task.
- In terms of the proposed model, there is no design that is specifically motivated by the unique challenges that short video pose. Instead, NAS is performed on top of existing state-of-the-art work. In other words, the model design is not well motivated but more through brute-force engineering upon existing models.
- It's unclear if the challenges mentioned in Figure 1 have been well tackled by the AutoShot model. How well does the best performing model do on the challenging cases described in Figure 1? Are there some qualitative demos on each type of challenge?
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. In my view the work is original as there is no existing dataset of such. The authors promise to release the code and data so hopefully the results are reproducible.
summary_of_the_review: Overall this is a good paper with a hopefully useful dataset that the video community can benefit from, especially due to the overwhelming trend of short and vertical videos. Although obtained by NAS on top of existing modules, the AutoShot model sets new standard of performance on multiple public benchmarks.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: For the task of shot boundary detection, this paper releases a new public dataset SHOT, which is complementary to existing related datasets. In addition, the model design is optimized by conducting neural architecture search in a search space encapsulating various advanced 3D ConvNets and Transformers. Experiments show the effectiveness of the proposed method.
strength_and_weaknesses: Strength:
1)	The newly released dataset SHOT is valuable, which will facilitate the development of community of short video understanding.
2)	The proposed neural architecture search based method is new for shot boundary detection, which outperforms previous SOTA approaches.

Weakness:
1)	It is difficult to understand for the readers without the neural architecture search background. It's technical contribution is limited. 
2)	The hyper-parameter sensitivity analysis is missing in the experiments.
3)	The dataset is relatively small, which consists of 853 videos. A larger dataset will be more meaningful. 

clarity,_quality,_novelty_and_reproducibility: The proposed method is novel and reproducible. However, some details of the method are not be clarified, as discussed in Weakness.
summary_of_the_review: This paper releases a new public dataset, as well as proposes an interesting method for shot boundary detection, which achieves SOTA. It is suggested to accept this paper.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies shot boundary detection, a long-standing problem, in a particular case: short-form videos. The challenges come from complicated shot gradual transitions, vertically ternary structured videos, and intra-shot change in virtual scenes. The authors collect a new public Short video sHot bOundary deTection dataset, named SHOT, consisting of 853 complete short videos and 11,606 shot annotations. Based on this dataset, the paper proposes to optimize the model design for video SBD, by conducting a neural architecture search in a search space encapsulating various advanced 3D ConvNets and Transformers. It achieves higher F1 scores than previous state-of-the-art approaches.
strength_and_weaknesses: Strength: 
- The paper shows its approach in detail, and the structure is formal. 
- The experiments are tested on different datasets.

Weaknesses: But I have the following significant concerns on quality, and novelty:

- Dataset construction needs several steps, including data collection, annotation, and verification. The authors may need to clarify that the collection process is not challenging. The data may come from web crawling or database accessing if they are from a video company. And at the same time, the copyright is not protected. How to handle and protect the copyright of video creators is questionable.

- The annotation is challenging since it requires frame-wise, but it is not reasonable. The annotation approach that uses a thumbnail cannot provide enough information for the annotators to well handle the challenging cases. Instead, the results may be similar to conventional methods that detect shots based on appearance. The details of the annotation are not clear in the supplementary. How to ensure quality? With double check or voting?

- The method lacks novelty that does not show much difference from the exiting NAS papers on classification. The challenges mentioned by authors are actually very good points for design, e.g., incorporating an adaptive sliding window to handle the gradual transition, or separately handling the background and subject for intra-shot changes, or using rules of thumb for ternary videos. A well-designed lightweight mixture-of-expert network may well handle the challenges. 

- The experiments are less interesting that cannot provide insights on the challenges mentioned by the authors, noting that the challenges are the motivation of the authors to bring out a new setting. And the qualitative results are misleading, and it is hard to tell the difference of different methods. And this also adds to my concerns about the quality of annotation.

Minor:
- What is the dataset scale comparison with the existing datasets?
clarity,_quality,_novelty_and_reproducibility: Please refer to the above for clarity, quality, and novelty. For reproducibility, I think a graduate student can reproduce it.

summary_of_the_review: Considering the novelty and quality of this paper, I think it does not reach the bar of the top-tier ICLR conference and vote for rejection.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Authors propose a new dataset for short video shot boundary detection (SHOT) and a corresponding baseline model obtained with neural architecture search. Authors argue that short videos are among the most popular media sources which renders the problem highly significant. The dataset consists of 853 short videos and approximately 11K shot annotations. The neural architecture for the baseline model is performed over a search space consisting of various 3D CNNs and Transformer models. The authors show that in this setting the model outperforms prior work in terms of the F1 score, both on SHOT and several existing datasets.

The reviewers appreciated the timeliness of this research given the popularity of short videos -- there is a clear need for new public benchmarks to facilitate development and evaluation of these models. However, the reviewers also identified several points for improvement. (1) Limited dataset size, hence limited diversity. (2) Model design seems to be closer to a brute-force approach combining existing models, rather than a well-motivated method for handling short videos. (3) Limited technical contribution and novelty. (4) Organisation and clarity of presentation. While some of these can be addressed in a revised version, the rebuttal didn't adequately address the main challenges and I will hence recommend rejection.
justification_for_why_not_higher_score: - Very limited dataset size
- Limited technical contribution and novelty
Decision: Reject