Generalized Video Moment Retrieval

Authors: You Qin, Qilong Wu, Yicong Li, Wei Ji, Li Li, Pengcheng Cai, Lina Wei, Roger Zimmermann
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: video moment retrieval, diverse query types, model versatility

Abstract:
In this paper, we introduce the Generalized Video Moment Retrieval (GVMR) framework, which extends traditional Video Moment Retrieval (VMR) to handle a wider range of query types. Unlike conventional VMR systems, which are often limited to simple, single-target queries, GVMR accommodates both non-target and multi-target queries. To support this expanded task, we present the NExT-VMR dataset, derived from the YFCC100M collection, featuring diverse query scenarios to enable more robust model evaluation.
Additionally, we propose BCANet, a transformer-based model incorporating the novel Boundary-aware Cross Attention (BCA) module. The BCA module enhances boundary detection and uses cross-attention to achieve a comprehensive understanding of video content in relation to queries. BCANet accurately predicts temporal video segments based on natural language descriptions, outperforming traditional models in both accuracy and adaptability. Our results demonstrate the potential of the GVMR framework, the NExT-VMR dataset, and BCANet to advance VMR systems, setting a new standard for future multimedia information retrieval research.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730271295535

thread_type:
official_review

<thread_object>
creator: reviewer_lxse

<content>
summary:
This paper introduces the Generalized Video Moment Retrieval (GVMR) framework, which expands traditional Video Moment Retrieval (VMR) to accommodate both non-target and multi-target queries, moving beyond the usual single-target focus. To support this, the authors present the NExT-VMR dataset, derived from the YFCC100M collection, featuring diverse query scenarios for robust model evaluation. They also propose BCANet, a transformer-based model that incorporates the Boundary-aware Cross Attention (BCA) module to enhance boundary detection and improve understanding of video content in relation to queries. Experiments demonstrate the potential of the NExT-VMR dataset and BCANet to advance VMR capabilities.

strengths:
- This paper introduces a new task called Generalized Video Moment Retrieval (GVMR), which is designed to adeptly handle the complexities of various query types, extending the traditional Video Moment Retrieval (VMR) paradigm to include both multiple-target and no-target queries.
- This paper presents a specialized dataset, NExT-VMR, which is meticulously constructed and analyzed from the YFCC100M dataset. Tailored specifically for GVMR, this dataset features a diverse range of query types, including one-to-multiple and no-target queries.
- This paper introduces a model called BCANet, aimed at improving performance on the challenging task of GVMR, where the model must accurately predict a variable number of temporal timestamps based on natural language descriptions.

weaknesses:
- Why does the paper utilize the YFCC100M dataset as the primary data source instead of existing VMR datasets such as TACoS, Charades-STA, MAD, or ActivityNet-Captions?
- From the statistics in Table 1, it can be seen that the average number of segments corresponding to a query is less than 2, which contradicts the paper's motivation for creating multiple-target queries. Additionally, the proportion of non-target queries is very low. Based on the dataset statistics, it is unclear where the challenges of this new dataset compared to existing ones lie.
- From Table 2, it can be seen that BCANet shows a significant advantage only in the single-moment retrieval task, while its performance in multi-target and non-target scenarios is not as pronounced. This raises questions about the overall effectiveness of the method.

questions:
- BCANet is a supervised learning framework. How does it address the issue of inconsistent target quantities for each query in a batch during the training process? Additionally, if a query corresponds to three moments, are all three segments' timestamp information used in the loss function calculation?
- For queries labeled as non-target, what information do they provide during training?
- Why does the paper utilize the YFCC100M dataset as the primary data source instead of existing VMR datasets such as TACoS, Charades-STA, MAD, or ActivityNet-Captions?
- From the statistics in Table 1, it can be seen that the average number of segments corresponding to a query is less than 2, which contradicts the paper's motivation for creating multiple-target queries. Additionally, the proportion of non-target queries is very low. Based on the dataset statistics, it is unclear where the challenges of this new dataset compared to existing ones lie.
- From Table 2, it can be seen that BCANet shows a significant advantage only in the single-moment retrieval task, while its performance in multi-target and non-target scenarios is not as pronounced. This raises questions about the overall effectiveness of the method.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730466394421

thread_type:
official_review

<thread_object>
creator: reviewer_4qqv

<content>
summary:
This paper introduces the Generalized Video Moment Retrieval framework, which expands traditional VMR to handle complex, non-target, and multi-target queries. To support this, the authors present the NExT-VMR dataset and BCANet, a transformer-based model with a Boundary-aware Cross Attention module that improves boundary detection and query understanding. 
The results show that GVMR, NExT-VMR, and BCANet significantly enhance VMR performance, setting a new benchmark for future research in multimedia information retrieval.

strengths:
The problem the authors solve is interesting.

This paper provides a good moment retrieval dataset.

The paper is clearly written and the method can be understood.

weaknesses:
- The NExT-VMR dataset is not large enough to be used for large-scale training. In addition, why not divide the test set on NExT-VMR for experiments?

- Figure 3 is confusing. There are many colored blocks around the Query-Text Cross Attention module. I don't understand what it wants to express? Lack of technical details.

questions:
- Can't the previous model achieve multi-target query well after training on the standard dataset? Lack of relevant experiments to prove the author's motivation.

- Can the model in this paper use a single query to query multiple related clips in the video? This is a more worthy of attention.

- YFCC100M is an excellent large-scale video dataset, but why did the author only select 9957 videos from YFCC100M? How was it selected?
- Will the author open source the dataset and code?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730530865848

thread_type:
official_review

<thread_object>
creator: reviewer_zk91

<content>
summary:
This paper introduces the Generalized Video Moment Retrieval (GVMR) framework, which extends traditional Video Moment Retrieval (VMR) to handle a wider range of query types. To support this expanded task, it presents the NExT-VMR dataset, derived from the YFCC100M collection, featuring diverse query scenarios to enable more robust model evaluation. Additionally, a novel transformer-based model named BCANet is proposed and achieves outperforms when compared with traditional VMR approaches.

strengths:
1. A novel GVMR framework is first introduced that significantly expands the scope of traditional VMR, catering to a wider range of query types and enhancing the model's practical utility.

2. A meticulously curated dataset, NExT-VMR, is presented. The dataset is specifically designed for GVMR task, which includes a variety of query types and scenarios, thereby enabling more robust and versatile model training and evaluation.

weaknesses:
1. As mentioned in the paper, "GVMR aspires to deliver a robust and flexible solution for video moment retrieval, significantly enhancing the model's interpretative prowess, precision, and retrieval efficiency." Would the enhancement of the model on interpretative prowess, precision, and retrieval efficiency be proved and presented in experiments?

2. More statistics of the proposed dataset are needed to be listed in Table 1, such as the entire/average lengths of videos and queries and the number of one-to-multi and no-target samples in the dataset.

3. Is it possible to demonstrate the evaluation of this method on traditional VMR datasets (e.g., Charades, ActivityNet, and TACoS) for one-to-one (different from the mentioned one-to-multi or no-target) VMR tasks? 

4. The BCANet pipeline presented in Figure 1 is confusing and difficult to interpret. The respective variable symbols should be labeled in Figure 1. Moreover, the processes for the BCL and QCA modules, along with query-text cross-attention, are also not clearly depicted in the figure. Moreover, what's the meaning of the "region" and "region features" in this paper? Dose the "region" mean a video clip?

5. The BCANet proposed in this paper does not seem to include specific designs for handling one-to-multi cases. Compared to traditional VMR methods, what advantages does BCANet offer in dealing with such data?

6. Since the retrieval speed is also an important metric in VMR, for example, [a] analyze the speed of various VMR approaches. I recommend the authors conducting speed analysis in this paper. In addition, is GVMR slower than traditional VMR?
[a] Fast video moment retrieval, ICCV 2021.

questions:
Please refer to the weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730942728996

thread_type:
official_review

<thread_object>
creator: reviewer_feek

<content>
summary:
This paper first introduce a novel GVMR framework that significantly expands the scope of traditional VMR, catering to a wider range of query types and enhancing the model’s practical nutility and  present a meticulously curated dataset. To deal with GVMR, they propose the BCANet model. Totally speaking, the contributions about extend task and new dataset is valuable.

strengths:
1. This paper introduces a extend task about TVG which covers more situations about video localiztion.
2. They build a new dataset for their introduced dataset which helps the deeper researches in future.
3. They propose a model based on LLM generate queries in fine-grain semantic, to benefit the video content understanding to the next stage

weaknesses:
1. The boundary awareness is a common question in this field and many works like[1] focus on this problem. The author didn't analyse the differences between them to show their novelty.


*[1]J. Wang, L. Ma, and W. Jiang, “Temporally grounding language queries in videos by contextual boundary-aware prediction,” in AAAI, 2020.

questions:
1. The model relies on the LLaVa to generate scene captions to divide the backgrounds and foregrounds. Will it brings noisy due to the diversity of generated samples?
2. Did you compare your model with the finetuinng model based on multimodal LLM ?
3. With using the LLM to generate captions , what about the time cost and memory usage?
4. Will the dataset be open source? If so ,it will be an important contribution.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731983968386

thread_type:
public_comment

<thread_object>
creator: ~luo_yinhang1

<content>
title:
Ethical Concerns Regarding License Compliance and Data Usage in Derived Datasets from YFCC100M

comment:
I have concerns regarding the ethical aspects of this paper's use of the YFCC100M dataset. The following points need clarification:

1. License Compliance: 
   - The YFCC100M dataset comprises various media files with differing licenses, yet the paper does not specify how the data was selected from the original dataset. There is no clear explanation of whether the selected data complies with the original licenses, particularly regarding permissions for modification and display.
   - For example, in the paper, some images are displayed (e.g., Figure 1, 2). It is unclear whether these images are permitted to be displayed under their respective licenses.

2. Webscope License Agreement:
   - The YFCC100M dataset is distributed under the Webscope License Agreement, which explicitly prohibits the redistribution of modified or derivative datasets. Relevant sections include:
     - Section 2.6: "You will not display, reproduce, transmit, distribute, sell, publish, provide to third parties, incorporate into third-party materials, or use the Data except as specifically permitted in this TOU."
     - Section 2.5: "You will not perform any analysis, reverse engineering or processing of the Data or any correlation with other data sources that could be used to determine or infer personally identifiable information of any Verizon Media user(s or that violates any applicable law in any applicable jurisdiction."
   - The paper mentions modifications (e.g., creating the NExT-VMR dataset) and displays content from the dataset, but it does not explain whether such actions align with the Webscope License Agreement or whether explicit authorization was obtained.

3. Community Contribution:
   - If the NExT-VMR dataset derived from YFCC100M cannot be redistributed due to license restrictions, the contribution to the community would be limited. This issue warrants further discussion and transparency in the paper.

The authors should address the following:
- Clearly state how the data was selected and whether the selection complies with the original licenses of YFCC100M.
- Confirm whether the modifications, analysis, and display of data align with the Webscope License Agreement or if explicit permission was obtained.
- Discuss the implications of potential license restrictions on the accessibility of the derived dataset and its contribution to the community.

Without addressing these points, there are significant ethical concerns regarding the use and redistribution of the data in this work.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732361274292

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Dataset Source and Licenses

comment:
# Data Source and License Compliance

The YFCC100M dataset comprises metadata for approximately 99.2 million photos and 0.8 million videos from Flickr, all shared under various Creative Commons licenses [1][2]. While access to the dataset through Yahoo Webscope is governed by the Webscope License Agreement, the original media files are hosted on Flickr under their respective Creative Commons licenses. This is clearly stated on the YFCC100M dataset homepage (<https://www.multimediacommons.org/>), and we directly crawled the data from Flickr. The origin description is as follows: 

> "The YFCC100M is the largest publicly and freely usable multimedia collection, containing the metadata of around 99.2 million photos and 0.8 million videos from Flickr, all of which were shared under one of the various Creative Commons licenses."

Furthermore, in the YFCC100M dataset papers (both versions [1][2]), there is no relevant description about dataset restrictions.

## Use of Creative Commons-Licensed Media

Our research utilized videos directly obtained from Flickr, strictly adhering to the terms of their respective Creative Commons licenses. These licenses permit the use and display of the media, provided that the conditions specified by each license are met, such as attribution and non-commercial use. The sections of the Webscope License Agreement, specifically Section 2.6 ("You will not display, reproduce…") and Section 2.5 ("You will not perform any analysis, reverse engineering or processing"), refer to restrictions related to Yahoo Webscope’s own proprietary data, not the data source we obtained from Flickr.

## Compliance with Licensing Terms

We did not use any additional annotations or metadata from the YFCC100M dataset that are restricted under the Webscope License Agreement. Our work involved only the media content available under Creative Commons licenses, ensuring full compliance with the permissions granted by the original content creators.

## Related Research Contributions

The YFCC100M dataset has served as the foundation for several impactful research contributions, further demonstrating its utility and importance in advancing the field. Notable examples include:

1. **"FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation," published in WACV 2021**, which created a face attribute dataset aimed at bias measurement and mitigation [3].
2. **"Towards Open-ended Visual Quality Comparison," published in ECCV 2024**, exploring visual quality assessment tasks [4].
3. **"Panoptic Video Scene Graph Generation," published in CVPR 2023**, focusing on generating structured representations for video understanding [5].

These works have similarly derived additional datasets and insights using YFCC100M as their starting point, aligning with the same ethical and licensing principles. It is precisely due to the open availability of YFCC100M that the dataset has been so successful and has garnered over 2,000 citations to date.

## Display of Images in the Paper

The images displayed in our paper (e.g., Figures 1 and 2) are used in accordance with the terms of their respective Creative Commons licenses. Similar to [3][4][5][6], we have ensured that we have the right to display these images for academic and research purposes, following all necessary attribution requirements.

## Contribution to the Community

Acknowledging the licensing restrictions, we will release only the video features extracted from the Creative Commons-licensed media and the annotations we have created ourselves. This approach allows us to contribute valuable resources to the community while respecting all legal and ethical guidelines.

## Ethical Considerations

Dear Yinhang,

As we have explained above, we have thoroughly evaluated the ethical aspects of our work and have consulted with the appropriate authorities to ensure compliance with all relevant policies and licenses. We are committed to upholding ethical standards in our research and fostering a responsible research environment.

Due to anonymous policy restrictions, we can only provide the information above to address your concerns. Additionally, we have submitted confidential comments to the Program Chairs and Area Chairs, and we believe there are no ethical concerns in our work.

Thank you again for your thoughtful feedback. We are open to further discussions to clarify any additional concerns and are committed to transparency in our research practices.

---

## References

[1] YFCC100M: The New Data in Multimedia Research. arXiv:1503.01817  
[2] YFCC100M: The New Data in Multimedia Research. Communications of the ACM, Volume 59, Issue 2. 2016  
[3] FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation. WACV 2021.  
[4] Towards Open-ended Visual Quality Comparison. ECCV 2024.  
[5] Panoptic Video Scene Graph Generation. CVPR 2023.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposed a new Generalized Video Moment Retrieval framework, which expands traditional VMR to handle complex, non-target, and multi-target queries. To facilitate research on this task,  a new dataset named NExT-VMR dataset, is derived from the YFCC100M collection, featuring diverse query scenarios to enable more robust model evaluation. The advantage of the method is that it defines a new task and establishes a relevant dataset, proposing a effective baseline model. The weakness is that some references, which were already published in some conference, were cited in arxiv format. Since all reviewers leaned to accept this paper, I recommend accept to it if there is not ethic problem.