Youku Dense Caption: A Large-scale Chinese Video Dense Caption Dataset and Benchmarks

Authors: Zixuan Xiong, Guangwei Xu, Wenkai Zhang, Yuan Miao, Xuan Wu, LinHai, Ruijie Guo, Hai-Tao Zheng
Primary area: datasets and benchmarks
Keywords: Chinese Video Datasets, Retrieval, Grounding, Generation

Abstract:
With the explosive growth of video content, video captions have emerged as a crucial tool for video comprehension, significantly enhancing the ability to understand and retrieve information from videos. However, most publicly available dense video captioning datasets are in English, resulting in a scarcity of large-scale and high-quality Chinese dense video captioning datasets. To address this gap within the Chinese community and to promote the advancement of Chinese multi-modal models, we develop the first, large-scale, and high-quality Chinese dense video captioning dataset, named Youku Dense Caption. This dataset is sourced from Youku, a prominent Chinese video-sharing website. Youku Dense Caption includes 31,466 complete short videos annotated by 311,921 Chinese captions. To the best of our knowledge, it is currently the largest publicly available dataset for fine-grained Chinese video descriptions. Additionally, we establish several benchmarks for Chinese video-language tasks based on the Youku Dense Caption, including retrieval, grounding, and generation tasks. Extensive experiments and evaluations are conducted on existing state-of-the-art multi-modal models, demonstrating the dataset's utility and the potential for further research.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1729399542085

thread_type:
official_review

<thread_object>
creator: reviewer_qgjf

<content>
summary:
The paper introduces Youku Dense Caption dataset, which is the largest publicly available Chinese dense video captioning dataset for now. The dataset is annotated by human to guarantee the quality of the dataset.
Building upon the proposed dataset, the paper also establishes benchmarks for video-language tasks.
The experiments demonstrate that existing state-of-the-art multi-modal models can benefit from this dataset.

strengths:
- The paper focuses on the topic of Chinese dense video captioning dataset, which is an interesting and under-explored research area.

weaknesses:
- Doubtful value of the proposed dataset
    - First, while the proposed dataset is claimed to be a dense video captioning dataset, the collection pipeline is similar to regular video captioning datasets, like HD-VILA-100M or Panda-70M, where a long video is first segmented into multiple clips and each clip is annotated with a caption. Could the authors provide more differences between the dataset collection pipelines of your dataset and a regular video-text dataset?
    - Second, it is unconvincing to state that "Chinese and English have significant linguistic differences, so a Chinese dataset is needed". I appreciate the authors show the errors of translation in line 211 and Section 3.2.2. However, I use ChatGPT to translate the provided samples and it can produce correct results in most of the cases. Take the leftmost sample in Figure 3 as example, I got this: "A group of motorcyclists is resting by the roadside, chatting." from ChatGPT, which is totally correct.

- Lack of necessary experiments: to evaluate the value of the proposed dataset, the authors need to train a model on different datasets and show that the one trained on the proposed dataset is more robust than the others. Such experiment should be conducted on different tasks, such as dense video generation, partially relevant video retrieval. However, none of this experiment is presented.

- It has been shown that long and detailed prompts are beneficial to various tasks, such as video generation. However, the caption annotations are short and less detailed, limiting the value of the dataset.

- For the scene change detection algorithm mentioned in lines 345~364, TransNet-v2 should be more robust than the adopted pixel-based algorithm.

questions:
- How is a long video segmented into multiple clips? What is the splitting criteria?

- Could you show more data samples in Youku Dense Caption dataset? It is helpful for checking the diversity, visual quality, and annotation quality  of the proposed dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730274628426

thread_type:
official_review

<thread_object>
creator: reviewer_va4c

<content>
summary:
This paper details a novel Chinese language based dense video caption dataset.

strengths:
+ This work collected a new Chinese based dense video captioning dataset (named Youku Dense Caption). The dataset has several benchmarks for Chinese video-language tasks, including retrieval, grounding, and generation tasks. 
+ This allow developer and researcher to train multmodal foundation model with a fair benchmark. 
+ This submission has validate the impact of large scale dataset on existing multimodal model. Providing empirical evidence of the advantage of rich data in the context of Chinese language.

weaknesses:
- The proposed dataset's video are evenly sampled from the Youku-mPLUG dataset based on dedicated (sub)categories. So the assumption is that the licensing should not be an issues. To properly handle the copyright concerns, please details the licensing terms for the Youku-mPLUG dataset, and discuss the coverage of usage right, redistribution policies, and any restriction.

questions:
- Has the author consider to release the English caption of the proposed dataset? In my opinion, this will broaden the impact of this dataset and benefit more downstream tasks. Please confirm if this is a a planned future work, as well as to discuss the plan for creating high-quality English translation of the captions. Further analysis on the English translation with open-source tool in Fig 1, it is clear that it requires professional proofreading or checked by someone (crowdsourcing?) who are fluent in both Chinese and English language.

- This paper should provide addition details related to the annotation progress. Please provide detilas on: 
1. The total number of human annotators involved.
2. The qualifications or expertise of the annotators (e.g., native Chinese speakers, etc.) and how are them recruited. 
3. The cost and time spent on annotation per video. 
4. The total number of annotations per annotator and the overall duration. 
5. Any quality control measures sued during the annotation process.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730292731033

thread_type:
official_review

<thread_object>
creator: reviewer_qncd

<content>
summary:
The paper presents Youku Dense Caption, the largest publicly available dataset for Chinese dense video captioning, comprising 31,466 videos annotated with 311,921 captions. Collected from the Youku video platform, this dataset addresses the lack of high-quality Chinese video captioning datasets and promotes advancements in Chinese multi-modal research. It provides benchmarks for key tasks such as retrieval, grounding, and generation, and extensive experiments demonstrate its effectiveness on state-of-the-art multi-modal models. The dataset’s scale and quality make it a valuable resource for future research in video-language understanding.

strengths:
Youku Dense Caption contains 31,466 short videos annotated with 311,921 captions, making it the largest dataset for fine-grained Chinese video descriptions.It addresses the scarcity of high-quality Chinese dense video captioning datasets, promoting advancements in Chinese multi-modal models and video-language research.The dataset establishes benchmarks for key video-language tasks such as retrieval, grounding, and generation, with extensive experiments demonstrating the utility of the dataset on state-of-the-art multi-modal models.

weaknesses:
1. Lack of Caption Diversity and Detail: In Figure 1, the captions for different segments of Video ID 1070344446 show high similarity with little distinction, lacking sufficient variability. Additionally, the descriptions are relatively simple and do not provide background information about the visual content.
2. Potential Hallucination in Captions: In the D. Implementation Details of Baselines section, the authors mention that they convert videos to 320p resolution and remove the audio component. However, in Figure 1, the second frame of Video ID 1192027222 shows the caption: “The old lady boasts that young women who work hard at chopping can do it.” It is difficult to determine solely from the visual content that the old lady is boasting, raising concerns about the potential for hallucinated captions, especially for those tied to audio-related information.

questions:
1. Vocabulary Statistics and Comparison: Can the authors provide vocabulary statistics and a comparison with other datasets? The captions appear to contain a high degree of repetition.
2. Threshold for Average Self-BLEU: Why was the threshold for Average Self-BLEU set to 0.15?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730595078616

thread_type:
official_review

<thread_object>
creator: reviewer_c5ms

<content>
summary:
This paper introduces Youku Dense Caption, a large-scale Chinese dense video captioning dataset. Dataset addresses the scarcity of high-quality Chinese video captioning resources, containing 31,466 short videos with 311,921 Chinese captions. A strategies is proposed to improve benchmark quality by filtering out redundant or low-quality annotations. The authors establish several benchmarks for Chinese video-language tasks and conduct extensive experiments demonstrating the dataset's utility and potential for research. They also discuss challenges related to the linguistic and cultural differences between Chinese and English video data.

strengths:
1. A Chinese video captioning dataset is proposed to fill the research gap in the Chinese community for video captioning data.
2. A embedding-based similarity and a Non-Maximum Suppression method is used to set up a Chinese PRVR benchmark that effectively reduces annotation redundancy.
3. The work reduces redundancy in video captioning and grounding by filtering out videos with high self-BLEU scores and minimal scene changes,  which is measured through color histogram correlation, ensuring a diverse and representative dataset.

weaknesses:
1. The statement in the section “Chinese Characteristics” seems unclear. The English translation of the so-called “fine-grained Chinese captions” could also serve as fine-grained English captions in an English-language context. For me, only the localized data part is valuable, as it highlights a major difference between Chinese and English video captions. Adding more data and statistics to support this distinction would strengthen the paper.
2. In the experiment, when translated back to Chinese. It's kind of blur that which attributes of Chinese dataset lead to the poor performance, the analysis failed to state clearly about the language differences between Chinese and English.
3. In ablation study, mixing of different datasets only strike a balance between different tasks but failed to achieve idealized performance across different tasks. And the best performance comes from larger data scale rather than data distribution and video-caption pair. 
4. Overall, the dataset serve as a valuable data source for Chinese community in video caption domain, but the value and key attributes of the dataset remain unclear and is not fully proved by the experiment.

questions:
see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733291489071

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All

comment:
We appreciate all the feedback and suggestions from the reviewers. We are pleased that the reviewers recognize our work and acknowledge our efforts to address their concerns.

--------------

Here, we would like to highlight and clarify the **Contributions of Youku Dense Caption**:

1. Why is proposing a new Chinese video captioning dataset a core contribution?
   - The Youku Dense Caption dataset addresses the lack of high-quality Chinese video captioning datasets, being the largest of its kind and promoting advancements in multi-modal models and video-language research.

2. How do the technical methods enhance dataset effectiveness and reduce redundancy?
   - By using embedding-based similarity, Non-Maximum Suppression, and algorithms like filtering videos with high self-BLEU scores and minimal scene changes, this work reduces redundancy in video captioning and grounding, ensuring a diverse and representative dataset.

3. How does the dataset advance research in video-language tasks?
   - The dataset provides benchmarks for tasks like retrieval, grounding, and generation, and experiments on different tasks and existing benchmarks show the advantages of our dataset for training multimodal models in Chinese.

--------------

We duly address the reviewers' concerns in the individual responses.

Here, we summarize the **Key Issues and Clarifications** for addressing the concerns in the revision.

  - Chinese Characteristics: Reviewers questioned the understanding of "Chinese characteristics" in our dataset. We addressed this by providing additional examples and explanations to highlight its significance. (Section 3.2.2)
  - Experimental Methods and Data Gain: Concerns were raised about the effectiveness of the experiments and the dataset's contribution to different tasks. We validated these contributions through ablation studies. (Section 6 Ablation Study)
  - Data Distribution and Validation: To address concerns about data distribution, we provided additional experiments to ensure the reliability of the results. (Appendix H)
  - Diversity and Hallucination Issues: Reviewers were concerned about diversity and hallucination in the dataset. We alleviated these concerns by providing more examples. (Appendix G)
  - Vocabulary Statistics: Reviewers requested detailed vocabulary diversity and generation quality analysis, which we provided. (Appendix E and Appendix F)

--------------

Finally, we sincerely appreciate reviewers for their valuable suggestions, which have significantly helped us enhance the quality of our work. Additionally, we regret that reviewer Qgjf did not have enough time to thoroughly review our manuscript and participate in the rebuttal discussion.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents Youku Dense Caption, a large-scale Chinese dense video captioning dataset. In the rebuttal, the authors effectively addressed several concerns, including clarifying the key value of the dataset of presenting “Chinese characteristics”, and resolving questions related to cross-dataset evaluation, copyright issues, dense captions and caption quality. As a result, all four reviewers unanimously recommended acceptance. The AC agrees and the authors should further revise the final version according to reviewers’ comments.