Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In this paper, the authors focus on moment retrieval and highlight detection tasks from videos given text queries. They present the QVHIGHLIGHTS dataset which includes 10,000 videos collected from Youtube. Each video is annotated with a query, time span of a moment, and saliency scores.
In addition, the authors propose an Encode-Decoder Transformer model (Moment-DETR) that predicts moments and scores in an end-to-end manner. 
The Moment-DETR model provides superior performance in comparison to previous methods on the QVHIGHLIGHTS dataset for moment temporal localization and highlight detection tasks.
main_review: - line 114: During data collection, how do you segment raw videos into short clips? Is it random or based on specific criteria?
- In table 4, can you explain why removing L1 or gIoU losses decrease the highlight detection performance?
- Can TVR and ANetCaptions datasets help in pre-training the MomentDetr model? (removing HD related losses)
- Although the Moment-DETR model performs well on QVHIGHLIGHTS dataset, there is no evidence that it will generalize to other datasets. For instance, how will it perform on the TVR dataset in comparison to SOTA?
- Authors extract video and text features using the CLIP model. Is there a way to use the nearest neighbor approach to evaluate moment retrieval (distance between text feature and clip features)? how well does it perform in comparison to Moment-DETR?


Typos:
- line 79: lsimiting -> limiting
limitations_and_societal_impact: The authors described the technical limitations but I couldn't find any information regarding the negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces a new dataset for video moment retrieval and highlight detection named QVHighlights. The dataset contains around 10K YouTube videos. Each video is annotated with a free form text query and (i) the time intervals in the video corresponding to the query (multiple intervals can match the query) and (ii) for each annotated time interval a *saliency* scores (from 1 (Very bad) to 5 (very good)) describing how relevant are the individual frames within an annotated clip are to the query. 

In addition to the dataset, the authors introduce a method named MomentDETR inspired from DETR in order to perform moment retrieval in their introduced dataset. The method performs better than existing techniques.
main_review: ## Strengths 

- The paper is clearly written
- The dataset is going to be useful to the community as it seems it is quite high quality (based on the description and the rater agreement). 
- I believe that the introduced method MomentDETR is a good extension to the DETR approach for moment retrieval in videos. 
- The pre-training strategy on weakly supervised data with ASR is interesting and leads to a substantial improvement.


## Weaknesses

Overall the paper is good but there a few points that could improve it further:

- **Weakly supervised pretraining**: The weakly supervised pretraining seems to bring a substantial gain in performance (Table 3). However from what I understood, only a small dataset is used for this pretraining (only 5.4K videos). Have you thought about using larger datasets for this pretraining? How would that impact the performance of the method? One could for example use the HowTo100M dataset for that [22]. I believe that a scale study would be very valuable there even if not scaling to the full HowTo100M dataset (e.g. providing the performance with only 2.5K videos and 10K videos to see the impact of increasing the size of the pretraining). The VLOGs dataset [a] could also be used for that purpose.

- **MomentDETR evaluation**: MomentDETR is an interesting method in itself. For that reason, I think it would be worthwhile to run MomentDETR on other moment retrieval benchmarks to show it is competitive on known benchmarks. I think one of the main issue of the paper is that it introduces a new dataset and a new method so it is hard to convince the reader that the method is indeed better than pas work such as [30,19,12,4,16] as we cannot compare direclty to some numbers provided in these papers. I do trust the authors for having run these baselines properly but I think that running the method on other datasets would increase the strength of the paper.

- **Precision about the saliency loss**: 
  - How are $t_{high}$ and $t_{low}$ defined based on the scores from 1 to 5?  
  - How important is the term using low/high versus the term using out/in. Could you provide an ablation about this in Table 4 (keeping all other elements i.e. L1/gIoU and CLS)? If the term low/high is indeed important that would justify more the need for annotating individual frames which is one of the specificity of the QVHighLights dataset.

- **Highlight and saliency score**: Some frames within positive annotated segments are annotated as `Very bad` (and `Bad`) for the given query. This seems to contradict that these frames are indeed positive for that query. In that case why not removing these frames all together from the annotation and create more moments? It would also be useful to provide the score distribution (% of frames that receive a score of 1,2,3,4,5...). 

- Minor typos:
  - L79: lsmiting -> limiting
  - L100: sailiency -> saliency
  - L326: detetcion -> detection

- Missing related work:

  - [a] From lifestyle vlogs to everyday interactions. Fouhey et al. CVPR18

## Overall assessment 

Overall the paper makes good contributions. However I believe that these contributions could be strengthen following the advices provided in the weaknesses section. For now I lean towards acceptance and I am ready to increase my score if some/all concerns are addressed during the rebuttal.


## Post rebuttal

After reading other reviews and the author's response I am happy to support acceptance of this paper. I really encourage the authors to include all the new ablations and set of experiments requested by the reviewers as they will make for a stronger paper, which in turn will be more influential in the community. 
limitations_and_societal_impact: The authors have not commented on the limitations and societal impact of their work.

Since the authors introduce a new dataset, I would recommend the authors to fill a datasheet for it (see [Datasheet for dataset, Gebru et al.](https://arxiv.org/abs/1803.09010))
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper collects the QVHIGHLIGHTS, a unified benchmark dataset for text-based moment retrieval and highlight detection in videos. It contains over 1000 Youtube videos with a) human-written free-form queries and their relevant moments, and b) five-point scale saliency scores for all query-relevant clips. Different from existing text-based moment retrieval datasets, QVHIGHLIGHTS has a smaller temporal bias on moments and allows a text query to match multiple disjoint moments. Compared to other highlight detection datasets, it presents temporally dense annotations dependent on queries. 
As a baseline, they propose a Moment-DETR model which can do the two tasks jointly.
main_review: Strength:
1. The dataset unifies two tasks based on text queries -- moment retrieval and highlight detection, and show how joint training can affect the performance of the baseline model on these two tasks respectively.
2.The collected dataset overcomes some of the limitations in other moment retrieval datasets by having less temporal bias and allowing a text query to match multiple disjoint moments.
3. The dataset accounts for the subjectiveness in highlight detection and provide dense annotations.
4. Good quality control through the qualification test and the verification between multiple workers.


Weakness:
1. Line 259-260 'we define a prediction to be positive if it has a high IoU with
one of the ground truth moments'.  It is not very clear here whether 'one of the ground truth moments' means one from 3 annotators or one from disjoint moments. 
2. For highlight detection: Line 262-263, 'we have ground-truth saliency scores from 3 users, we evaluate performance against each then take the average'. Does it mean the model's performance is dependent on the inter-user agreement?
3. Missing baseline experiments/ablations: 
 - The results from using only CLIP and Slow-Fast networks as visual encoder, as in how important the temporal information is in the two tasks.
 - Performance on moment retrieval after the model is pre-trained via ASR, to see how much improvement has supervised training brought.
 - Evaluation of SoTA models on moment retrieval which have official implementations online: 2D-TAN[1], DRN[2], CBP[3]
4. Performance of the baseline model:
 - Since one advantage of the dataset is having queries matching disjoint moments, how well can the baseline model spotting all the disjoint moments given a query, is there a metric to evaluate it?
 - Table 1 shows the dataset has a wide range of granularity in the verbs (e.g, cut and report) and nouns(dog and weather), does the performance of the baseline model varies by the granularity?
5. While the highlights are labelled on a five-point scale, is that only the 'very good' ones are used in training and evaluation?
6. Related work: [4] uses a DETR-like architecture for temporal detection, and treat the task as a set-prediction task.

[1] Learning 2D Temporal Localization Networks for Moment Localization with Natural Language. Zhang et al. AAAI 2020.
[2] Local-Global Video-Text Interactions for Temporal Grounding. Mun et al. CVPR 2020.
[3] Temporally Grounding Language Queries in Videos by Contextual Boundary-aware Prediction. Wang et al. AAAI 2020
[4] Activity Graph Transformer for Temporal Action Localization. Nawhal et al. CVPR 2021.

Summary:

The QVHIGHLIGHTS dataset addresses some limitations in current moment retrieval/highlight detection datasets. It is presented in a clear way and will be beneficial to future research in these areas. However, some detailed analysis on the dataset is missing, and more baseline experiments are needed to verify the design choice in the baseline architecture. 

limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper introduces a new dataset using language queries to localize moments in videos. The paper also introduces a DETR-like baseline for this task. Several existing models and the baseline are run on the proposed dataset. I'm concerned the contributions will not be significant, the dataset is small (10k videos) and the baseline is an adaptation of existing methods to videos.
main_review: Overall the paper is fairly well written, and easy to understand. The task of localizing moments in videos from language queries is interesting, important and would be quite valuable to solve. The code and dataset annotations are provided, which should help in reproducing and extending the work.

However, the paper has a few weakness currently:

(1) Throughout the paper, it is mentioned that the dataset is diverse, but I do not see any metrics of diversity of the dataset. It seems misleading to claim diversity without providing any support for it. Also, the paper does not have any mention of ethical concerns or societal impact. Given that such a dataset was created, especially using personal VLOG style videos, this seems like it should have some discussion.

(2) The dataset is also fairly small by current standards, 10,000 videos is not many.

(3) The proposed baseline approach is fairly straightforward adaptation of existing approaches, like DETR to videos. There is some value in seeing it work and providing a good baseline for the new dataset. It would also be good to compare the proposed baseline to previous works on other, existing datasets to help make comparisons.

In the checklist:
"Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [Yes] See Section 3.1."

I do not see any estimated payments, wage, or time spent on the task in Section 3.1 or the paper or supplementary materials.
limitations_and_societal_impact: There is some discussion on the limitations of the proposed pretraining, but otherwise there is no discussion on the limitations of the approach or dataset and no meaningful discussions on the societal impacts of the dataset.
review_title: null

Meta-review: The authors responded fully to the reviwers' concerns (new training, new baselines ...) and constructively to the ethics reviews. Following the rebuttal, all reviewers recommend acceptance.

Of course, the authors should update the paper and dataset as stated.

Congratulations!
Decision: Accept (Poster)