Youku Dense Caption: A Large-scale Chinese Video Dense Caption Dataset and Benchmarks
Authors: Zixuan Xiong
Guangwei Xu
Wenkai Zhang
Yuan Miao
Xuan Wu
LinHai
Ruijie Guo
Hai-Tao Zheng
Venue: ICLR 2025 Poster (2025)
Keywords: Chinese Video Datasets
Retrieval
Grounding
Generation
Primary area: datasets and benchmarks

Abstract: With the explosive growth of video content, video captions have emerged as a crucial tool for video comprehension, significantly enhancing the ability to understand and retrieve information from videos. However, most publicly available dense video captioning datasets are in English, resulting in a scarcity of large-scale and high-quality Chinese dense video captioning datasets. To address this gap within the Chinese community and to promote the advancement of Chinese multi-modal models, we develop the first, large-scale, and high-quality Chinese dense video captioning dataset, named Youku Dense Caption. This dataset is sourced from Youku, a prominent Chinese video-sharing website. Youku Dense Caption includes 31,466 complete short videos annotated by 311,921 Chinese captions. To the best of our knowledge, it is currently the largest publicly available dataset for fine-grained Chinese video descriptions. Additionally, we establish several benchmarks for Chinese video-language tasks based on the Youku Dense Caption, including retrieval, grounding, and generation tasks. Extensive experiments and evaluations are conducted on existing state-of-the-art multi-modal models, demonstrating the dataset's utility and the potential for further research.

Published as a conference paper at ICLR 2025
YOUKU DENSE CAPTION: A LARGE-SCALE CHINESE VIDEO DENSE CAPTION DATASET AND BENCHMARKS
Zixuan Xong!.3 Guangwei Xu
Wenkai Zhang Yuan Miao
Xuan Wu²
Hai Lin
Ruijie Guo Hai-Tao Zhengl,24 1,24
1 Shenzhen International Graduate School, Tsinghua University
*Pengcheng Laboratory *Alibaba Cloud Computing *Alibaba Group 1ex220mails.
edu.cn zhangwk0106@@maii. com
kunka. wwweealbbaaainneee.. com (miaoyuan.my my, wx1 93834 (0333336 salibba-iine com
ngyygmecutiook. com
rui riej www.allloaaaeeeee innc com zheng. haitaalst. t tignggua...... inghua.edu.co .edu.cn
ABSTRACT
With the explosive growth of video content, video captions have emerged as a crucial tool for video comprehension, significantly enhancing the ability to derstand and retrieve information from videos. However, most publicly available dense video captioning datasets are in English, resulting in scarcity of large- scale and high-quality Chinese dense video captioning datasets. To address this gap within the Chinese community and to promote the advancement of Chinese modti models, we develop the first, large-scale, and high-quality Chinese dense video captioning dataset, named Youku Dense Caption. This dataset is sourced from Youku, prominent Chinese video-sharing website. Youku Dense Caption includes 1,466 complete short videos annotated by 311, $19,921 Chinese cap- tions. To the best of our knowledge, it is currently the largest publicly available dataset for fine-grained Chinese video descriptions. Additionally, we establish several benchmarks for Chinese video-language tasks based the Youku Dense Caption, including retrieval, grounding, and generation tasks. Extensive experi- ments and evaluations are conducted on existing state-of-the-arl multi-modal mod- els, demonstrating the dataset's 's utility and the potential for further research. The dataset is publicly available http://github.com//pennnStt -alel-AAAYoooouuo- -Dense- Caption.
INTRODUCTION
The proliferation of the internet and mobile devices has precipitated an exponential increase in the volume of video content (Wang Gao 2020). Across social media platforms, video-sharing web- sites, and online education and entertainment applications. video has emerged as a predominant medium for information dissemination and consumption (Levy) 2021). However, this surge in video content presents significant challenge in efficiently understanding and retrieving information from videos (Dong et al., 2022b). Among the various auxiliary tools available, video captions have gar- nered considerable attention for their pivotal role in enhancing video content comprehension and facilitating information retrieval (Gernsbacher 2013 Perego et al. 2010).
Currently, the majority of publicly available dense video captioning datasets are predominantly in English, resulting in a notable scarcity of resources for non-English languages, particularly Chi- nese. This linguistic gap only constrains the user experience for Chinese-speaking audiences but also impedes the development and optimization of multi-modal models tailored to Chinese video content (L et al. 2019 Singh et al. 2020)..
To address this critical gap and foster the advancement of Chinese multi-modal models, we intro- duce the Youku Dense Caption dataset, the first large-scale, high-quality Chinese dense video cap-
Project Lead.
Coorrispnndngg author; cheen..hattadee @ ww..iinguu..ed....

---

Published as a conference paper at ICLR 2025
Table 1: Statistics of Youku Dense Caption and its comparison with existing video-language datasets. Avg. Len denotes the average segment length, referring to the mean length of the seg- ments designated by each annotation.
Dalasets
Type
Lamguage
Videos
# Text
Avg. Len
Domain
MSR-VTT Xuu et al. 2016
video captioning
EN
7k
200k
14.8s
open
MSVD Chon Dolan 2011
video captioning
2k
85k
open
VATEX Wang
9019b
video cuptioning
ENICH
41k
826k
10s
open
Youku-mPLUG
al cf 2023
video captioning
CN
80k
80k
54.2s
open
ActivityNer Captions (Krishna et al. 2017
dense vidco captioning
EN
20k
100k
36s
open
YouCook2 Zhou et al. 2018
dense vidco captioning
2k
15k
7.7s
cook
VITT (Huang et al. 2020
dense video captioning
8k
open
Youku Dense Caption
dense video captioning
CN
31k
311k
8.1s
open
tioning dataset, meticulously designed to meet the needs of Chinese video content comprehension and information retrieval. This dataset sourced from Youku, one of China's ' leading video-sharing platforms, and comprises 31,466 complete short videos annotated with 311,921 Chinese captions. This makes it the largest and most detailed publicly available dataset for fine-grained descriptions of Chinese video content, thereby providing substantial resource for Chinese video-language pro- cessing research.
In addition to offering comprehensive dataset, we establish several benchmarks for Chinese video- language tasks based on the Youku Dense Caption dataset. These tasks include video retrieval, grounding. and caption generation. These benchmarks not only provide rigorous setup pipeline for the objective evaluation of existing multi-modal models, but also guide future research and develop- ment directions in this domain.
To validate the utility of the Youku Dense Caption dataset, we conduct extensive experiments and evaluations using state-of-the-art multi-modal models. The results of these experiments demonstrate the dataset's significant impact on improving model performance, including video retrieval and cap- tion generation. Through this research, we underscore the potential of the Youku Dense Caption dataset in advancing the field of Chinese video-language development.
Our main contributions are follows:
We introduce the Youku Dense Caption dataset, the largest and fully human-annotated Chinese dense video captioning dataset, with 31,466 short videos and 311,921 Chinese captions.
We establish several benchmarks for Chinese video-language tasks, including video retrieval, grounding, and generation, providing standard evaluation metries for multi-modal models.
We validate the effectiveness of the dataset through extensive experiments, demonstrating its sig- model generation and retrieval.
nificant impact on enhancing the performance of
2 RELATED WORK
Dense Video Captioning Dataset Dense video captioning (DVC) datasets represent special-
ized subset of video-captioning resources, distinguishing themselves from traditional datasets by focusing on the nuanced task of narrating multiple events within video rather than generating
single overarching description. The DVC datasets discussed, such as MSR-VTT (Xu et al.) 2016), MSVD (Chen Dolan 2011). YouCook2 [Zhou al. 2018). VATEX (Wang et al. 2019b). Activ- ityNet Captions (Krishna et al., 2017), YouMakeup (Wang et al. 2019a). and (iTT(Huang 2020), provide rich annotations
(Huang et al.) include event-level details and diverse categories, enabling mod- els to capture and describe events sentence by sentence. These datasets vary in size, scope, and linguistic diversity. For instance, MSR-VTT comprises around 10,000 videos across various do- mains, while VATEX offers multilingual annotations for over 41,000 clips. Moreover, datasets like YouCook2 (Zhou et al. 2018) highlight domain specificity, focusing on cooking videos, thus facil- itating tasks like recipe generation and video summarization. Collectively, these resources advance research in video understanding, natural language processing, and the development sophisticated DVC models capable of effective event localization and description.
2

---

Published as a conference paper at ICLR 2025
1070344466
Timedamps
6.09-17:37
17.37-2681
26
5442-63188
63.
SE de 1A 18 Ealii 30
Be 1e 11 x cougie arrigeesoo rectonrpiien Righnay
MIR A 96 WE Thare an sutancycis Encooghh mosonai
Amotaaions
80 30
buach bngghtio
home's oosettio
The lanabo ce autangck caviit
The ridor currebiie loreating en
thei maviing tut
road and aaling
Viduo
1192027222
Timestamps
3.71-9.59
1492-17.68
24.72-27.96
95.05-2.95
50. 00.19-56.77
56 56-15-9.54
A BE A
R: A 1509 S ALAEEE 58 Mick
Amotations
and Cww.. sttre
young Kimee shlo work
The Jayy hoats But
5F & A.E
woren dinging stgge
youug exeme andi
A A 9999999 vommar with camting
Figure I: Sample videos and annotations from the Youku Dense Caption dataset. We randomly se- lect two videos and sample six frames from each, displaying their respective timestamps and corre- sponding Chinese text annotations. Additionally, we provide English translations of the annotations using the open-source tool OPUS-MT (Tiedemann et al. 2023 Tiedemann & TThottingal 2020
Video-Language Downstream Benchmarks The related works (Wang et al. 2024b Xu et al. 2023) on vidco-language downstream benchmarks primarily focus on evaluating and enhancing the performance of tasks related to video content description and retrieval, encompassing various inter- national challenges and competitions. For instance, the Large Scale Movie Description Challenge (LSMDC) (Rohrbach et al.) 2017) is based on movie video dataset and aims to generate both sin- gle and multiple sentences that describe video segments, utilizing human and automated evaluation methods to assess the quality of the generated text. Similarly, the Video-to-Languag challenge, based on the MSR-VTT dataset, requires participants to generate complete sentences that describe video content, combining both automated and manual evaluations for scoring. Additionally, the TREC Video Retrieval Evaluation (TRECVID) (Awad et al., 2023) explores the matching and gen- eration tasks between video and text descriptions. Finally, the VATEX Video Description Challenge focuses multilingual video descriptions, encouraging the use of video information as context to enhance translation quality. These benchmark studies promote advancements in video content un- derstanding and natural language description by proposing diverse tasks and evaluation standards.
YOUKU DENSE CAPTION DATASET
To address the gap fine-grained annotated datasets in the Chinese community, we present the first large-scale dense video captioning dataset with detailed Chinese annotations. The dataset comprises 31,466 videos, segmented into 311,921 clips, with a cumulative duration of 748.96 hours. Each video has average duration of 85.68 seconds, while the average clip length is 8.1 seconds. On average, each video contains 9.9 annotations, with each annotation consisting of 17.9 characters. The entire process, from data cleaning to the generation of Chinese annotations, was meticulously carried out manually to ensure the highest quality of data.
3.1 DATA SOURCES
The dataset is constructed to meet the following requirements: 1) it should cover the most common video themes; 2) the duration of the videos should be no less than one minute to ensure that the content is meaningful. Based on these requirements, the raw videos the dense caption dataset are evenly sampled from the Youku-mPI.UG dataset based on 11 major categories and 84 subcategories,
3

---

Published as a conference paper at ICLR 2025
Video Duration Distribution
Clip Duration Distribution
CONL
FILLIPE
PIN
Caption Length Distribution
Category Distribation
Caption Count Distribution Per Video
THC
+0.0
TIL
Figure 2: Statistics of Youku Dense Caption. Please note that in A, 8%, A represents the exact quantity, while represents the corresponding percentage.
with this categorization derived from the video tags. The Youku-mPLUG dataset contains 10 million videos collected according to strict safety, diversity, and quality criteria. In addition, annotators are instructed to exclude any videos that presented difficulties in annotating, such as those that are unclear, split screens, illustrations, image carousels, meaningless effects, ensuring that the videos in the dataset are of high quality and rich in event information. More annotation details are in the Appendix.
3.2 STATISTICS AND FEATURES
We now present the statistics of the Youku Dense Caption dataset and compare them with other similar datasets. Fig. summarizes the most notable aspects of the Youku Dense Caption dataset concerning both its visual and linguistic content.
3.2. BASIC STATISTICS
The Youku Dense Caption dataset is characterized by its diversity and richness, making it a valuable resource for research in video understanding. caption generation, and natural language processing.
Extensive Category Coverage. The dataset includes 11 coarse-grained categories (e.g., Life, Va- riety Shows, Finance and Economies, Agriculture, Children, Fashion and Entertainment) and 84 fine-grained subcategories (e.g., Film and Television includes movies, TV shows, anime, etc.). This wide range ensures comprehensive coverage of video types across open domains. Balanced Video Duration. 99% of the videos have durations between to 2 minutes, ensuring rich content and feasi- bility for fine-grained annotations. The duration segments are relatively balanced, each comprising about 20% of the dataset, providing good representation of different video lengths. Short Clip Focus. Predominantly featuring clips ranging from short video content analysis and processing. The uniform distribution of clip durations aids in model generalization across different clip lengths. Consistent Caption Lengths. Captions are primarily between 10 to 26 tokens, with the majority in the 18 to 22 token range. This consistency supports dense captioning and short text generation tasks, helping maintain stable model performance during training. Rich Caption Information. Each video contains 3 to 33 captions, providing ample data for dense captioning and video content understanding research. This layered distribution facilitates studies video comprehension and generation under varying caption densities.
seconds, the dataset is well-suited for
4

---

Published as a conference paper at ICLR 2025
Vid: 1089492007
Vid: 1272141709
Vid: 1182715104
Vid: 1255983397
Vid: 1108754691
Vid: 1490996201
ES: 0.16-6.59
0.00-18.34
29.48-31446
1S: 60.20- 100.87
TS: 26.69-30.55
37.04-38.68
NRW BE AI
6 222 8 de 21 4
+ 12:0000
M Eu & 16 03 A yellir
L 03 f
LARAF
weaning
23 ko
ras weriing kay proparing
On sage, vouties datto grorting ondryaa
homeddy
mun catting Zeagel will scbiass
A fex chite and bakiin Nowe decentionn lands coggatddate
Figure 3: Examples of culture unique to China the dataset. Vid indicates video ID, and 7S stands for timestamp. The red text highlights instances of unique and iconic terms in Chinese culture.
3.2.2 CHINESE CHARACTERISTICS
Firstly, there exist significant differences in cultural background and contextual understanding be- tween Chinese and English-speaking communities. Unique elements of Chinese culture, such as Festival moon gazing, and Chinese New Year greetings, are rarely
the tea ceremony. Mid-Autuse encountered in public English datasets but are quite common in Chinese contexts. This cultural dis- parity is evident not only in the distribution of annotated texts but also in the content of videos. For detailed cultural explanations, please refer to Appendix
As emphasized by the widely used Chinese CLIP (Yang al. 2022). bridging the gap in annotations and video content caused by cultural differences is the core value of our dataset. Understanding and bridging these cultural differences By highlighting these unique cultural elements, our dataset not only better reflects the actual needs of Chinese users but also provides greater accuracy and cultural relevance in the distribution of annotations and video content.
crucial for accurate translation and content representation.
4 BENCHMARK SETUP
4.1 PARTIALLY RELEVANT VIDEO RETRIEVAL
An untrimmed video is considered partially relevant to a given textual query if includes a segment that pertains to the query. Partially Relevant Video Retrieval (PRVR) (Dong et al. 2022a) aims to identify and retrieve these partially relevant videos from a vast collection of untrimmed videos. Existing English benchmarks, such as TV Show Retrieval (Lei et al.) 2020), ActivityNet Captions, and Charades-STA (Gao et al. (2017), directly use segment-level annotated texts as queries to search within video. This approach overlooks the possibility of similar segments and annotations exist- ing across different videos, which leads to these similar annotations matching with multiple videos rather than being specific repetitive retrieval processes during evaluation, exacerbating the evaluation bias of the benchmark. To address this issue, we post-proces, the data to establish a Chinese PRVR benchmark. This bench- mark ensures that a single query corresponds to multiple videos without repetitive query retrieval, thereby enabling a more debiased evaluation.
single video. Furthermore, these similar annotations cause nearly
First, we extract the segment-level annotation and video ID pairs to form the pairs [4, v.]. Then, we aggregate similar texts using the saobb--emeddiigg model (Huang et al. 2024 [Snn et al.) 2020). which has achieved state-of-the-art results in several Chinese retrieval tasks (Muennighoff et al.) 2023). For each annotation text t, we calculate its embedding and then create a set T; of all annota- tion texts with embedding similarity greater than 90%, along with corresponding set V; of video IDs. Thus, for each annotation text f; there is a corresponding set V,, indicating that all videos in V; contain segments related to f.-
In the second step, we focus on removing duplicate annotation texts to avoid redundancy. We employ method similar Mon-Maximum Suppression (NMS), commonly used in object detection tasks.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1 Cross-Video PRVR Benchmark Setup
1: Input: Dataset D {(t,, u1), (tz, vz).
(t),
(tg, 2/y J. is segment-level annotation.
2: Output: Benchmark B {14, Vi), .., (l,, Vi)), V42 is a set of video ids.
3: for each annotation text do
4: (4) = Text Encoder((:).
5: Create set T; of all annotation texts with embedding similarity 0.9 to
T, {ty sim(embed(tt)) embed(t,)) 0.9}
6: Create the corresponding set Vi of video IDs from pairs (4z, U1) where 1, E Tj:
Vi = {uy (1z, v2) E {tt, 2:)3 and t2 E Ti}
7: end for
8: for each pair of sets (V, Vi) do
9:
if len( > 1 and len 1 > then
10:
Calculate Intersection over Union (IoU):
V. V MOVI
[VU,V, Vg)
11:
if (V. Vi) >> 0.7 then
Remove (4,, V2) len( len(V)), else remove (1,, Vi)..
13:
end if
14:
end
15:
end for
16: Output: Remaining pairs (1,, Vi).
First, we calculate the Intersection over Union (IoU) between each pair of sets V; and Vj. The IoU is defined as the size of the intersection divided by the size of the union of the two sets:
If the IoU between Vi and Vj exceeds 0.7, we consider the retrieval process for t, and t, to be redundant. To resolve this redundancy, we compare the sizes of the sets V, and VG. If len(),) > (V,), we retain the larger set VA and remove the annotation Ij along with its corresponding set Vj, This iterative process ensures that we retain the most representative and comprehensive annotation sets, thereby reducing redundancy.
As result of these steps, build Chinese PRVR benchmark based on Youku Dense Caption, which consists 28,988 queries and 3,185 target videos, with an average of 4.96 related videos per query. We set two evaluation metrics: is the Top-k Accuracy, where retrieving any of the related videos within the top results considered meeting the requirements, and the other the Mean Average Precision, which measures the proportion of related videos in the top len(V,) results.
4.2 GROUNDING AND GENERATION
In the tasks of temporal grounding and caption generation, existing annotations of text and video segments face two main issues. Firstly, the annotated text for certain videos may contain a large amount of repetitive information due the lack of event changes. Such annotations are usually ambiguous and do not provide informative descriptions of the video segments. Secondly, the scene changes within video segments may not be significant. Even though the video is divided into differ- ent timestamps, the internal scene changes within the segments are minimal, resulting in repetitive and monotonous annotations. This monotony reduces both the diversity and objectivity of the eval- uation, making these video segments unsuitable for use evaluation data in tasks like temporal grounding and generation.
To address these issues, propose two strategies to improve the quality of benchmark datasets: First, we calculate the internal consistency of annotated texts. Specifically, we calculate the self- BLEU (Zhu et al. 2018) value for all annotated texts within each video. If the self-BLEU value of a video exceeds 0.15, the internal consistency of the annotated texts is considered too high, and the video is subsequently filtered out.

---

Published as a conference paper at ICLR 2025
The self-BLEU value for an annotated text is calculated as follows:
Self-BLEU(s:) = BLEU((s, {81, 82, 8i- 884 8it1, .,
(I)
where 8; is the annotated text being evaluated, and {51, 82......... 8:-1, 1 ,8) represents the set of all other annotated texts within the same video, excluding 8;.
The overall self-BLEU for video is then calculated the average of the self-BLEU values for all annotated texts 8y in that video:
Self-BLEU 1 Σ Self (elf-BLEU(s:)
where n is the total number of annotated texts within the video.
Second, we use the color histogram correlation of video segments to assess the degree of scene change between video segments. We perform color histogram analysis on each video segment and calculate the histogram correlation between adjacent segments. Specifically, the formula for caleu- lating histogram correlation (Correlation) is as follows:
(E (H2(() 122)
Correlation H1, H2)
(3)
(1) H1 H1) Em= (12 H12
where H1 and H2 represent the color histograms of two video segments, (1) and H2(1) represent the value of the ith bin in the histograms, and H, and H2 represent the mean values of H and H2, respectively. The mean values are calculated as follows:
H1 (1) H2
H2(1)
(4)
the color histogram correlation of a video segment exceeds 0.5, the scene change in the video considered insignificant, and the video is subsequently filtered out. In the end, obtain 1,872 videos and a total of 20,099 annotations, averaging 10.73 annotations per video.
The evaluation metrics are based existing work (Soldnn et al. 2022, Pan al. 2023 For grounding tasks, we use the R R and IoU-M calculation methods, which assess the top-N accuracy under the condition that the IoU is greater than M for the retrieved timestamps. For generation tasks, mainstream evaluation metrics such as BLEU (Papineni et al. 2002), METEOR (Lavie Agarwal 2007), CIDEr (Vedantam et al. 2015), and ROUGE-L. (Lin 2004 used for comprehensive assessment of the generated text. In addition, we also include the Bert score (Zhang et al., (020) measure the semantic similarity of the generated text to alleviate the evaluation bias caused by different generation styles.
EXPERIMENTS
In this section, we comprehensively evaluate several state-of-the-art video multi-modal large models on three benchmarks within the Youku Dense Caption dataset. We present more detailed introduc- tions to the baselines and implementation details in the Appendix.
Retrieval Performance In this experiment, we select three categories of models: video retrieval models, video generation models, and CLIP models. The rationale behind choosing these three types to comprehensively evaluate the performance of different multi-modal large models on Chinese video datasets and understand the impact of language differences. Given that the Youku dense caption dataset is in Chinese, and most existing large-scale video models, such as Intern Video (Wang et al. 2022; 2024c) and ViCLIP (Wang al. 2024b), are primarily trained on English corpora, translate the annotations into English using OPUS-MT to conduct bilingual evaluations in both Chinese and English.

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation results of the retrieval benchmark. We evaluate the existing various video models, image-text models, and generative models using only zero-shot methods. For specific ex- perimental details, please refer to the Appendix.
text2vddoo
video2text
Models
Language
Top
Top
Top
Top Acc
Top Ace
Top Ace
VICLIP eWang
2024b
English
26.78
47.53
56.33
56.38
76.95
84.39
Intern Intern Video2 Wiin al 1024
English
26.56
47.46
57.17
47.87
70.21
79.07
VICLIP
Chinese
0.91
4.71
7.92
2.12
8.86
12.41
InternVideo2
Chinese
1.31
3,90
6.94
1.77
4.60
7.80
mPLUG-OwB (Ye et
2024
Chinese
2.04
6.57
10.55
0.35
2,48
3.19
CLIPRadfond al 222
English
1.05
4.34
8.18
2.12
9.92
13.82
DFNSB-CLIP
2024
English
15.49
16.31
36.87
47.51
Chinese-C1.IP Tang el
0022
Chinese
31.78
55.49
65.72
41.13
64.53
70,92
For the video retrieval models, when comparing the retrieval performance ViCLIP and Intern- Video2 on Chinese and English datasets, it is evident that these models perform poorly on Chinese video retrieval tasks. This underperformance primarily due to the lack of training on Chinese corpora.
Regarding the video generation model, mPLUUGOWl1, which only supports multi-modal generation tasks, adapted it for the retrieval task by performing pooling the token embeddings encoded by representation. Despite this adaptation, mPLUG-Owl does not
its model to serve as its exhibit effective retrieval performance in its encoded embeddings. This is because mPLUG-Ow13 lacks specific retrieval training, which is crucial for such tasks.
Lastly, for the CLIP models, since there parison against English video large models like ViCLIP, we compare the retrieval performance of Chinese and English CLIP models of the same size. The Chinese-encoded model shows significant advantage over the English model when evaluated on Chinese and translated English datasets. This finding indirectly highlights the language differences between Chinese and English, emphasizing the importance of language-specific training for optimal performance.
native Chinese video large models available for com-
Overall, these experiments underscore the necessity of training multimodal models on diverse lin- guistic datasets to ensure robust performance across different languages.
Grounding Performance the grounding task, due the lack of temporally related tasks during the pre-training phase of the pre-trained multi-modal large model (Feng et al. 2023), multi-modal large models cannot be directly applied to downstream tasks such as temporal localization. There- fore, following Wang et al. (204c), we use these pre-trained models as feature extractors to encode multi-modal features and input them to Moment-DETR. (Lei al. 2021 for training, resulting in the Tab. 3
Table 3: Evaluation results of the grounding benchmark. In the evaluation, use the same- dimensional CLIP and Chinese CLIP for comparison to ensure the capacity of the temporal lo- calization model remains consistent.
Features
Dim
Language
RI @0.5
R1@0.7
mAP@0.5
mAP@0.75
mAP@avg
CLIP
512
English English
6.38
2.16
11.70
3.24
4.35
VICLIP
768
6.23
2.17
11.40
3.31
4.30
Chinese CLIP
512
Chinese
13.89
4.95
20.10
5.91
7.76
Due to the lack of Chinese-supporter video-language encoding models, we only employ VICLIP and CLIIPttpe models that perform well retrieval tasks. can be observed that, under the same encoding dimensions, Chinese- CLIC still demonstrates greater advantage compared to CLIP.
Generation Performance For generative tasks, most existing multi-modal large models rely on pre-trained large language models, which generally support multilingual generation. Therefore, do not perform additional language processing for this task. For different models, we use the same Chinese prompt Describe this video in one sentence..
8

---

Published as a conference paper at ICLR 2025
Table 4: Evaluation results of the generation benchmark. We directly segment the video using timestamps and generate descriptions for the video segments using these models.
Models
Vision Encoder
Language Decoder
BLEU-4
METEOR
Rouge-L
CIDEr
Bert Score
Intern Videe22ihe MiniCPM-V-2.6
Interviden2- 1B SigLip-400M Intern- VTT-6B
Mistral-78
0.35
6.92
7.25
4.97
58.69
Qwen2-7B
0.57
12.86
12.14
3.66
62.72
VL2288VL2-8B
Internl.22.5775
1.09
13.62
13.20
9.19
65.34
Swen2-VL-7BB-Iasee
DFN
Qwen2-7B
1.51
13.84
15.10
19.32
66.68
As shown in Tab. VnterVVL2-o 8B (Chen et al. 2023 Jiang et al. 2023), MiniCPM-V-24 V-2.6 Yao
2024 Zhai et al. 2023 Yang al. 2024) and Wiwen2-V--7B-Insse (Bai et al. 2023 Wang et al.
et al. 2024a) exhibit significant advantages video captioning. These models have undergone extensive training for image-text generation, which enhances their performance in this domain. In video processing, large models typically perform frame extraction, converting videos into multiple images for captioning tasks. This approach allows video captioning effectively leverage the strong image-text captioning capabilities of these models.
ABLATION STUDY
In this section, we experimentally evaluate the quality of our dataset as training data and how can enhance the model's performance on existing benchmarks within the community. Specifically, we fine-tune multi-modal models such as VICLIP and Qwen2-VL on datasets with Chinese video annotations, including VATEX and Youku-mPLUG. Additionally, we introduce proposed Youku Dense Caption (YDC) dataset as extra training data during the fine-tuning process and evaluate the performance the corresponding benchmarks to demonstrate the effectiveness of our dataset for training.
Improving Generation Performance with YDC We select the VATEX dataset with Chinese an- notations for auxiliary validation because it offers semantically similar annotations for each video. Furthermore, due to stylistic differences across various domains in generation tasks, we eval- uate both the VATEX validation set and the generation benchmark on YDC, which are of nearly the same scale. This dual evaluation aims to assess the cross-domait generalization capabilities of the training results.
Table 5: Experiments on achieving data augmentation in generation tasks using YDC data.
Training Sct
VATEX Val
Generation Benchmark on YDC
% VATEX
% YDC
BLEU-4
METEOR
ROUGE-L
CIDEr
BLEU-
METEOR
ROUGE-L
CIDEr
0%
0%
16.00
22.52
39.01
33,06
0.94
9.39
12.57
15.87
100%
0%
28.16
30.17
48.05
53.12
2.41
14.06
18.36
26.88
50%
50%
27.71
29.81
47.72
51.52
4.71
17.31
25.23
42.41
0%
100%
22.77
26.50
43.74
35.64
4.86
17.50
25.51
43.06
100%
50%
28.42
30.14
47.98
52.82
4.63
17.34
25.28
41.51
100%
100%
29.57
30.49
48.53
54.06
4.91
17.54
25.68
44,32
We conduct LoRA fine-tuning (Hu et al. 2022) based on the Qwen2-VL-2B model and compared different experimental results by adjusting the proportion of training data sources. Specifically, on one hand, we control the total amount of training data to remain roughly consistent and adjust the proportion of the VATEX and Youku datasets for training. On the other hand, we gradually add the YDC dataset while utilizing the full VATEX dataset to observe the data augmentation effect. Note that the VATEX training data contains 250k annotated texts, while the training set filtered and divided from the Youku Dense Caption for the generation task contains 278k video-text pairs, with similar data volumes.
During the experiments, we find that models trained only on VATEX or only on Youku Dense Caption did not achieve ideal results the evaluation set of the other domain. This indicates that there are significant differences between data from different domains, and training data from a single source cannot achieve generalization across different domains.

---

Published as a conference paper at ICLR 2025
To overcome this limitation, attempt to mix the training data of VATEX and Youku Dense Cap- tion. The results show that as the proportion of YDC training data increased, the Qwen2- VL model eventually achieved state-of-the-an of-the-ar results both evaluation sets. This demonstrates that our pro- posed YDC dataset, similar to the VATEX dataset, can bring significant data augmentation effects.
Additionally, we provided additional test results on the youku-mplug validation set under the same training set scale. The results demonstrate that, between datasets of comparable size, training on the Youku Dense Caption dataset achieves superior performance compared to Vatex. Additionally, to mitigate potential effects of data distribution bias, we evaluated the model on the Youku-mplug dataset, which represents different distribution from the training data.
Table Experiments on out-of-distribution (OOI) performance.
Training Data
Youku-mPLUG
BLEU-4
METEOR
ROUGE-L
CIDEr
Vatex
Validation Validation
3.08
12.77
19.53
29.53
Youku Dense Caption
3.46
13.54
22.76
31.55
Improving Retrieval Performance with YDC Here, we explore the contribution of the YDC data to the video-text retrieval task, using the Youku-mPLUG dataset as the base data. To ensure no overlap between Youku-mPLUG and YDC, we deduplicate the Youku-mPLUG data using file hashing and several video features. We select ViCLIP for fine-tuning, which achieves state-of-the- art results in English. The Youku-mPLUUG retrieval training data contains 36k video-text pairs, while the YDC dataset's 's retrieval training data amounts to 278k pairs. Therefore, 13% of the YDC training data in our experiments is equivalent in size to the Youku-mPLUG training data.
Table 7: Experiments achieving data augmentation in retrieval tasks using YDC data.
Training Set
Testing Set
video2text
tex12video
% Youku-mPLUG
% YDC
Youku-mPLUG
R1
R5
R10
R1
R5
RI0
0%
0%
Validation
1.03
2.94
4.78
0.40
1.32
2.24
100%
0%
Validation
2.42
9.86
16.55
1.90
8.36
14.12
0%
13%
Validation
0.46
2.30
4.09
0.34
1.55
3.17
0%
100%
Validation
3.46
11.18
17.30
3.46
11.36
17.07
100%
13%
Validation
2.99
9.91
16.03
2.24
8.24
13.26
100%
100%
Validation 5.88
17.12
24.62
5.47
15.62
23.47
From Tab. [ we can see that when fine-tuning with YDC training data of the same scale, the perfor- mance the Youku-mPLUG validation set decreases due to the out-of-domain distribution. How- ever, as the amount of YDC data increases, the generalization gains are significantly greater than the results obtained by training solely on Youku-mPLUUG This further demonstrates that the contribu- tion of the proposed YDC data can enhance the performance of retrieval tasks.
7 CONCLUSION
In this paper, we address the significant gap in Chinese dense video captioning resources by in- troducing the Youku Dense Caption dataset, the first large-scale, high-quality Chinese dense video captioning dataset. Comprising 31,466 short videos annotated with 311,921 Chinese captions, it is the largest and most detailed publicly available dataset for fine-grained descriptions of Chinese video content. Extensive experiments and evaluations using state-of-the-art multi-modal models demon- strate the dataset's utility and impact on improving model performance. Additionally, our work shows that incorporating the Youku Dense Caption dataset supplementary training data signifi- cantly enhances model performance. This dataset not only provides a critical resource for Chinese video-lan video-language processing research but also encourages further advancements in this field. We hope this work inspires more scholars to contribute to developing Chinese lideo-language processing technologies.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This research is supported by National Natural Science Foundation of China (Grant No.62276154), Research Center for Computer Network (Shenzhen) Ministry of Educa- tion, the Natural Science Foundation of Guangdong Province (Grant No.2023A15150122 and 4403000240031100001777 No.JCYJ202100241200 2033,,
Basic Research Fund of Shenzhen City (Grant
1111202408111220000 and 01172202200111113000000 the Major Key Project of PCL for Experiments and Applications (PCL2023A09 and PCL2022A05), Alibaba Research Intern Program.
ETHICS STATEMENT
The Youku Dense Caption dataset primarily features concepts and language expressions that are relevant at the time of its creation. As language naturally evolves with human activities, our dataset may not include emerging concepts, new words, or contemporary expressions in the future. The authors will handle the long-term maintenance of the Youku Dense Caption dataset and the bench- marks evaluated in our paper. The dataset will be available on Modelscope, with Alibaba Cloud providing backend support for the download service. To ensure stability, will regularly check the dataset URLs and promptly address any broken links, Our datasets are released under the Creative Commons Coniiiinnnnnnmmiiciiaaaaaaaaaa
4.0 International Public License ("CC BY-NC- SA 4.0"), along with additional terms outlined in this document. Users must agree to this license when downloading or using the datasets from our website.
REFERENCES
George Awad, Keith Curtis, Asad A. Butt, Jonathan G. Fiscus, Afzal Godil, Yooyoung Lee, Andrew Delgado, Eliot Godard, Lukas L. Diduch, Jeffrey Liu. Yvette Graham. and Georges Quénot. An overview on the evaluated video retrieval tasks at TRECVID 2022. CoRR, abs/2306. 13118, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, local- ization, text reading, and beyond. arXiv preprint arXiv:2308. 12966, 2023.
David L. Chen and William B. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, pp. 190-200. The Association for Computer Linguistics, 2011.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. CoRR, abs/2312. 2023.
Jianfeng Dong, Xianke Chen, Minsong Zhang, Xun Yang, Shujie Chen, Xirong Li, and Xun Wang. Partially relevant video retrieval. In ACM Multimedia, pp. 246-257. ACM, 2022a.
Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang, Xun Wang, and Meng Wang. Dual encoding for video retrieval by text. IEEE Trans. Pattern Anal. Mach. Intell., 44(8):4065-4000 2022b.
Alex Fang, Albin Madappally Jose, Amit Jain, Ludwig Schmidt, Alexander T. Toshev, and Vaishaal Shankar. Data filtering networks. In ICLR. OpenReview.net, 2024.
Wei Feng. Xin Wang, Hong Chen, Zeyang Zhang, Zihan Song, Yuwei Zhou, and Wenwu Zhu. LLM4VG: large language models evaluation for video grounding. CoRR, abs/2312,142066 2023.
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. TALL: temporal activity localization via
language query. In ICCV, pp. 5277-5285. IEEE Computer Society, 2017.
Morton Ann Gernsbacher Video captions benefit everyone. Policy insights from the behavioral and brain sciences, 2(1):195-00, 2015.
11

---

Published as a conference paper at ICLR 2025
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR. OpenRe- view.net, 2022.
Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, and Radu Soricut. Multimodal pretraining for dense video captioning. In AACL/IJCLP,, pp. 470-490. Association for Computational Linguis- tics, 2020.
Junqin Huang, Zhongjie Hu, Zihao Jing, Mengya Gao, and Yichao Wu. Piccolo2: General text embedding with multi-task hybrid loss training. CoRR, abs/2405.069 2024.
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anee Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. CoRR, abs/231 2023.
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. ICCV, pp. 706-715, IEEE Computer Society, 2017.
Alon Lavie and Abhaya Agarwal. METEOR: an automatic metric for MT evaluation with high levels of correlation with human judgments. In WMT@ACI, pp. 228-231. Association for Com- putational Linguistics, 2007.
Jie Lei, Licheng Yu, Tamara L. Berg, and Mohit Bansal. TVR: A large-scale dataset for video- subtitle moment retrieval. In ECCV (27), volume 12366 of Lecture Notes in Computer Science, pp. 447-463. Springer, 2020.
Jie Lei, Tamara L. Berg, and Mohit Bansal. Qvhighlights: Detecting moments and highlights in videos via natural language queries. CoRR, abs/2107.09609, 2021.
Ro'ee Levy. Social media, news consumption, and polarization: Evidence from a field experiment. American economic review, 111(3):831-87) 2021.
Sheng Li, Zhiqiang Tao, Kang Li, and Yun Fu. Visual to text: Survey of image and video captioning. IEEE Trans. Emerg. Top. Comput. Intell., 344)2297-31, 2019.
Chin Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL httpst:::: http://:///aaaaaooooooo...r..ooe
-0013
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: massive text embed- ding benchmark. In EACL, pp. 2006-2029, Association for Computational Linguistics, 2023.
Yulin Pan, Xiangteng He, Biao Gong, Yiliang Lv, Yujun Shen, Yuxin Peng, and Deli Zhao. Scanning only once: An end-to-end framework for fast temporal grounding in long videos. In ICCV, pp- 13721-13731. IEEE, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic evaluation of machine translation. In ACL, pp. 311-318. ACL, 2002.
Elisa Perego, Fabio Del Missier, Marco Porta, and Mauro Mosconi. The cognitive effectiveness of subtitle processing. Media psychology, 13(3):243-272, 2010.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139 of Proceedings of Machine Learning Research, pp. 8748-88888 8763. PMLR, 2021.
Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Joseph Pal, Hugo Larochelle, Aaron C. Courville, and Bernt Schicle. Movie description. Int. J. Comput. Vis., 123 (1):94-120, 2017.
Alok Singh, Thoudam Doren Singh, and Sivaji Bandyopadhyay. A comprehensive review on recent methods and challenges of video description. CoRR, 1t.s/2111..455/ 2020.
12

---

Published as a conference paper at ICLR 2025
Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba Heilbron, Chen Zhao, Silvio Giancola, and Bernard Ghanem. MAD: A scalable dataset for language grounding in videos from movie audio descriptions. In CVPR, pp. 5016-5025. IEEE, 2022.
Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In CVPR, pp. 6397-6406. Computer Vision Foundation IEEE, 2020.
Jörg Tiedemann and Santhosh Thottingal. OPUS-MT Building open translation services for World. In Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT), Lisbon, Portugal, 2020.
Jörg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Arne Grönroos, Tommi Nieminen, Alessandro Raganato Yves Scherrer, Raul Vazquez, and Sami Virpioja. De- mocratizing neural machine translation with OPUS-MT. Language Resources and Evaluation, (58):713-755, 2023. ISSN 1574-0218. doi: 10. 10.1007/s10579
Ramakrishna Vedantam, C. Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In CVPR, pp. 4566-4577 IEEE Computer Society, 2015.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint 1rXiv:2409.12191, 2024a.
Weiying Wang, Yongcheng Wang, Shizhe Chen, and Qin Jin. Youmakeup: A large-scale domain- specific multimodal dataset for fine-grained semantic comprehension. FMNIP//JCNIPP (I), pp. 5132-5142. Association for Computational Linguistics, 2019a.
Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for video-and- lannauugee research. In ICCV, pp- 4580-4590. IEEE, 2019b.
Xiwen Wang and Longxiang Gao. When SG Meets Industry 4.0. Springer, 2020.
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint 2r11:22212.03111, 2022.
Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma. Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, Ping Luo, Ziwei Liu, Yali Wang. Limin Wang, and Yu Qiao. Internvid: A large-scale video-text dataset for multimodal understanding and generation. In ICLR. OpenRe- view.net, 2024b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, Yansong Shi, Tianxiang Jiang, Songze Li, Hongjie Zhang, Yifei Huang, Qiao, Yali Wang, and Limin Wang. Internvideo2: Scaling video foundation models for multi- modal video understanding. CoRR, ab5/2403.15377. 2024c.
Haiyang Xu, Qinghao Ye, Xuan Wu, Ming Yan, Yuan Miao, Jiabo Ye, Guohai Xu, Anwen Hu, Yaya Shi, Guangwei Xu, Chenliang Li, Qi Qian, Maofei Que, Ji Zhang, Xiao Zeng, and Fei Huang. Youku-mplug: A 10 million large-scale chinese video-language dataset for pre-training and benchmarks. CoRR, abs/2306.04362. 2023.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. MSR-VTT: A large video description dataset for bridging video and language. In CVPR, pp. 5288-5296. IEEE Computer Society, 2016.
An Yang. Junshu Pan, Junyang Lin, Rui Men, Yichang Zhang, Jingren Zhou, and Chang Zhou. Chinese CLIP: contrastive vision-language pretraining in chinese. CoRR, abs/2211.00 1.01335, 2022.
13

---

Published as a conference paper at ICLR 2025
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang. Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report. CoRR, abs/2407.10671 2024.
Yuan Yao, Tianyu Yu, Ao Zhang. Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A GPT-4V level MLLM on your phone. CoRR, 0bs/2408.01800 2024.
Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mplug-ow13: Towards long image-sequence understanding in multi- large language models. CoRR, abs/2400.04840, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- praitngi In ICCV, pp. 11941-11952. IEEE, 2023.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evalu- ating text generation with BERT, In ICLR. OpenReview.net, 2020.
Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web instructional videos. In AAAI, pp. 7590-7598. AAAI Press, 2018.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang. and Yong Yu. Texygen: A benchmarking platform for text generation models. In SIGIR, pp. 1097-1100. ACM, 2018.
A ANNOTATION GUIDELINES
Objective The goal is to create detailed and coherent descriptions for the main scenes presented the provided video, including specific start and end timestamps for each description. The final annotation files are released in a video format.
Description Structure Each description must adhere the following structural components:
(1) Subject: Use general identifiers to describe subjects, avoiding specific names. Acceptable terms include "man", "woman", "young man", "boy", "child", "girl", "doll", "elderly person", and "groom". Avoid proper nouns or distinct identifiers, such as "Yang Mi", "Cangnan Mountain Im- mortal", or "gangster". (2) Description of the Subject: Include adjectives that pertain the subject's color, size, state, facial expressions, and emotions. (3) Description of Actions (Optional): Describe the actions performed by the subject, indicating their nature and purpose. (4) Description of the Scene (Optional): Specify the type of location, such "school", "plaza", "playground", "shop- ping mall", while avoiding specific site identifiers like "Mingzhu elementary school" or "Beijing city".
Annotation Requirements (1) Each description sentence must contain a minimum of ten words and consist of at least three sentences for a video. (2) Descriptions should provide specific details for each component (subject, action, and scene) and articulate events occurring in the video while maintaining grammatical and semantic coherence. (3) Common referential terms (e.g., "he", "she") may be used to reference previously introduced subjects. (4) A coherent narrative that encapsulates the primary content of the video should be produced, along with descriptions of noteworthy but non- primary scenes. (5) Annotations for overlapping video timestamps may be permissible. If multiple scenes present in single video, they should be described separately.
14

---

Published as a conference paper at ICLR 2025
Annotation Process (1) Complete Viewing: View the entire video to grasp the overall content. (2) Drafting Descriptions: After viewing, create comprehensive deseriptions based on the identified scenes. Each description should be coherent and reflect the content accurately. (3) Formatting Descriptions: Convert the drafted descriptions into the specified format: (description, [start time (s), end time (s))) corresponding to the actual playback positions.
Cleanup Some segments of the video may contain nonsensical content that precludes valid de- scriptions, such lengthy passages of text, meaningless advertisements, or "oral broadcasting" videos. In these instances, annotations for these clips should be omitted. Furthermore, we have ex- cluded videos lacking explicit category tags, these videos often exhibit ambiguous classifications or consist of composite footage from multiple genres. Additionally, issues are identified in the anno- tations, such as excessively brief annotation texts or an insufficient number of annotations per video. These instances of low-quality annotations, along with the associated videos, are also removed from the dataset.
Timeline The annotation process is expected to take approximately two months. This includes: (I) One round of initial marking by a single annotator. (2) quality check conducted by a second annotator. (3) A final review to ensure completeness and accuracy.
By adhering to these guidelines, annotators ensure that the descriptions produced are both insightful and reflective of the video's content, enhancing its utility for review and analysis.
Details Our annotation team consisted of annotation leader and 10 annotators. The leader was responsible for aligning annotation standards, conducting trial annotations, training annotators, and quality control. The annotators focused on specific annotation tasks and data revisions.
We engaged professional data annotation agency. The annotation leader had at least 3 years of experience in managing annotation tasks, while the annotators had a minimum of 3 months of expe- rience in CV-related data annotation. All team members were native Chinese speakers.
The annotation cost was 0.63 RMB per valid clip caption, with total video costs varying based on the number of valid slices. The entire process took months, with efficiency improving over time as annotators became more proficient.
While we don't have detailed information about the internal labor distribution within the annota- tion agency, we understand from our interactions with the annotation leader that the workload was generally evenly distributed among annotators.
The annotation leader conducted quality checks on 10% of each batch of annotations. A 97% pass rate was required for batch approval; otherwise, the entire batch was returned for re-annotation. In the early stages, we also performed secondary reviews of the data to ensure alignment with annota- tion standards.
B DETAILED DATA STATISTICS
The specific number of categories and subcategories for the entire Youku Dense Caption dataset is shown in the Tab. There are a total 31, 31,466 videos, categorized into 11 major categories and 84 subcategories.
When dividing benchmark data and non- benchmark data, we fixed the random seed to 42 and ran- domly selected 10% of the video IDs from the Youku Dense Caption dataset as the source for creating benchmark data. This subset consists of 38185 videos and their corresponding 31,553 anno- tations. The remaining 90% of the data, comprising 28,281 videos and 280,368 annotations, is used as the source for training data.
During the quality filtering process for the retrieval benchmark, the text2video part was filtered from the original 31 31,553 texts down to 28,98 988 texts for retrieval evaluation, with an average of 4.96 related video IDs per text. For the video2text part, For the retrieval task's training data, we did not perform any quality filtering and used the original single text-video pairs to form the training data.
simply converted the corresponding relationships.
15

---

Published as a conference paper at ICLR 2025
Table 8: Dataset Overview by Category and Subcategory
Agpfoattue (230
2313
2.725
Childrrn (3P
2/06
Agricktrnl Tichewdage clc" (t/1))
1.137
Coepashendte Spors (f (1
315
Pureeting and Culdeire FF
60
Spurts Neas (FST
Earty Entckood Ecacatin 19
Agreshoon Pndiciss (74000
Chillee (2
Corqutiio Spent C2C00 RD
787
Chikkoeis Variey Sh (U422)
Backetul (2C00
Mariecaed (8FEE
7.54
(1425)
Calirr 2
Wi
Ausmatica (9:20)
Cruncly Themer (B0 (0/70888
1000
Mas Vikeo (4V)
<2
Manic Clio (1U5
2,003
Counaly Skip 01032 T)
700
Spmeszrel Perferzaano (1
60
Mavic Exina (TLF2006
Compacicniiic Canady (967000)
TV Skan Extra (1 20/21)))
Ramal Bilandin (2.0 Riff)
149
Daxusataz Clip (11 (1300
2.134
Fankl Eibe eien (8
S6
Doxmastay Eeazs
Cleen (898)
Laagage Leaving (8/15)11))
31
(260006
Hunney (7
Tradiieul Technooogy (110544
Chisoe Daase (1R))
04
Heakh and Wellace (f-- 18:P)
10:
Modiaa and Heath
d
Hair Soyteg (7)
Oigital Nev Madia (0F9)
1JC4
Ballmoors (41
32.
Horra Deoor (2
Regutittattin (P-
Orata Dance (1530)
76
Red Form and Reewwrtte (28 (Ltb)
65
Squar Donce (999)
60
Traal
Mideen Thawe (8,1
Ammiigs (10 90)
Bodiice (1-47)
155
Aurreviiiar (7t.%)
Beeas 69:30
Thendy Taye (855)
Nail 41
(:853
Chagpsanneeed Drace (BBbbi))
55
Real (E St)
1,00
Sacd Daxc (9r20)
72
Canxe ued Waiiggtn (8:4,31
Rap
Cuto Rakes Daly (Ph10)
1.572
Mask Now (811) (111)
67
Varatt Shews (0925)
1,341
Elnance (8115)
1
Vansry Show Clape (8977)18))
1.192
Emma
Vanaty Show Provows and Newe (5922 (FFR)
Reaace Baoiac (141577 (2)
215
Reasdill Experts (0117-2000
315
For the generation and localization tasks' benchmark quality filtering, we first applied a threshold filtering method based on the average self- value of annotated texts within single video, reducing the original 3,185 vidcos to 2,746 videos. Then, by applying a threshold based on the color histogram correlation of video segments corresponding to timestamps, we further filtered down to 1,872 videos as the benchmark data, with an average of 10.7 text annotations per video. For the training data of the localization and generation tasks, filtered the original training set to 16120 videos, with an average of 10.82 annotations per video.
Finally, facilitate benchmark evaluation, we additionally selected 282 videos from the original unfiltered 3, 185 videos' test data to create a mini test set, using the same post-processing method to produce the mini benchmark. The experimental results in this paper are all derived from the mini benchmark evaluations.
C PIPELINE OF BENCHMARK SETUP.
We randomly select 10% of the original dataset using a fixed seed to create the benchmark. These data annotations, if used as training data or for other data augmentation purposes, do not require post-proct post-processing due to their high quality and rich variety of categories. However, as benchmark data, they may contain some redundant data, leading to biased and less objective evaluations. Ad- ditionally, the annotation style or video type may not be suitable for certain tasks. For example, in temporal localization tasks, if video's scene changes are minimal or the annotation texts are quite similar and lack distinguishing information, it can be difficult even for humans to differentiate. In fact, when given a video segment and its corresponding text annotation, the data relationship is rea- sonable. However, such data is only suitable for data augmentation and not for evaluation due to its characteristic of being reasonable in forward annotation but ambiguous in reverse inference. There- fore, we adopt the data post-pre processing method shown in the figure below to establish objective and comprehensive benchmark.
D IMPLEMENTATION DETAILS OF BASELINES
In the initial data processing, the videos include both landscape and portrait orientations. We first resize them to either 360x640 or 640x360 dimensions, and standardize the frame rate 15 FPS.
16

---

Published as a conference paper at ICLR 2025
Siepl: Generate Embbaldings
Vidoos
Vidao Clips
Tex Entooding
Land
testekkngg
Teass
Timosamps
Kalalld
Canar Coats Pali
White Coflectnn)
Video Text Pairs
Step2: Similar Text Each Embedkling
Averaee
Axeriga
Step3: Comesponding Video Groups
Groundin and Genevation Benchwark
(edeal vide? (dbe3) veli prdes viout vides.
Cliys
Histograns
Cross- Video Retricval Benchmark Construction Pipeline
Vndro-Vidco Generation Grounding Benchmark Construction Pipeline
Figure 4: Pipeline of benchmark setup.
Then, using the processing script from the somethingsomettligg-22 dataset, we convert the vidcos to a 320p format and remove the audio component.
ViCLIP VICLIP does not have many version distinctions; we use default official demo code to uniformly extract 8 frames from the video for encoding through ViCLIP. For the localization benchmark evaluation, since moment-detr requires features for every two seconds of the video, we segment the video into two-second clips and then encode these segments using ViCLIP to serve as the features for these two-second clips.
InternVideo2 In the generation task, we use the Intern Video2-8B-Chal version, which consists of the visual encoding model from Intern Video2-1B and the language decoding model from Mistral- 7B. For video processing, we uniformly extract 8 frames from the video as input, and the Chinese prompt input the model is Use sentence describe the video. Please answer in Chinese In the retrieval task, we use the Intern Videood-o2-steee 1b-224p-f4 model, which is trained in the second stage of Intern Video2. The input frame limit 4, we uniformly extract four frames from the video for encoding and retrieval.
InternVL2 In the generation benchmark evaluation, tion. We uniformly extract frames from the video as video features and use the Chinese prompt Use sentence describe the video. for caption generation.
the official default demo for genera-
For the grounding benchmark evaluation, we use the base-scale 512-dimensional CLIP and Chinese CLIP models. In the retrieval benchmark evaluation, we use the huge-scale 1024- dimensional models. Since CLIP does not have huge version, we use the huge version of DFN- CLIP for our experiments. For feature extraction in the localization benchmark, we select the middle frame of each two-second video segment to encode and represent the features that segment.
CLIP
mPLUG-OW13 As a multi-modal generative model, mPLUG-OW13 first encodes visual data into embeddings, then inputs both the text prompt and visual embeddings into the language model si- multaneously. In the retrieval benchmark evaluation, we use the visual embeddings encoded by the model as visual features, and input the text prompt into the language model obtain the output hidden vectors. We then perform mean pooling on these hidden vectors to obtain the text features. Note that the number of frames extracted from the video is 16.
17

---

Published as a conference paper at ICLR 2025
Qwen2-VL In the generation benchmark evaluation, Qwen2-VL VL dynamically extracts video frames, meaning the longer the video, the more frames are extracted. This is made possible by its unique architecture, which supports dynamic resolution imput.
Table Comparison of word frequency. We uniformly selected 20 words from the top 1000 high- frequency words before and after filtering for display.
Original
Original Freq.
Processed Word
Processed Freq.
Rank
Annotation Word
25
i (One side)
0.48611
3 4
(Yellow)
0,45717
75
59
1:
(Gentleman)
0.19027
(Suit)
0.18388
125
7-fH (Finger)
0.10943
(Lion)
0.11035
175
2A H&R (Suit)
0.07742
C (A handful)
0.08013
225
THE (Outside)
0.06525
f (One unit)
0.06621
275
НА (Inside the room)
0.05288
O (Delicious)
0.05445
325
Pa mi
(Dark)
0.04419
$7 (Jeans)
0.04496
375
EK (Sweater)
0.03774
1n X (Join)
0.03805
425
HA (Quiet)
0.03356
# (Arena)
0.03444
475
カ* 2 (Walk back and forth)
0.02899
K (Long dress)
0.03001
525
(On the desk)
0.02641
# (Nod)
0.02681
575
Whith
(By the wall)
0.02383
D (Sunshine)
0.02424
625
#
(Electric vehicle)
0.02164
10 (Angry)
0.02248
675
(Tear with teeth)
0.02023
X/T (Hall)
0.02063
725
7 (Grizzled)
0,01881
7E2X (Pattern)
0.01928
775
7J
(Scissors)
0.01752
12 (Tire)
0.01836
825
(Work clothes)
0.01642
E (Straight)
0.01702
875
# 4
(Driving)
0.01546
10 (Slowly)
0.01588
925
K (Khaki)
0.01443
T (Top of head)
0.01475
975
f (Brush)
0.01353
T (Exhibition hall)
0.01403
MiniCPM-V In the generation experiments with MiniCPM-V, uniformly extract 64 frames from the video for generation. The Chinese prompt used Briefly describe this video in one sentence because it always tends generate overly long video descriptions.
Training Details In the data augmentation experiments for the generation task, train the data for epoch using the default fine-tuning parameters of the Swift framework. The learning rate is set to le-4, and the batch size is set to 16. The training is conducted eight A800 GPUs, resulting in an effective batch size of 128.
In the data augmentation experiments for the retrieval task, train the data for 10 epochs with learning rate of 8e-4. The batch size per GPU is set to 256, and the training is conducted on eight A800 GPUs, resulting in an effective batch size of 2048.
E VOCABULARY STATISTICS
Note that Section 4. we employed methods for data filtering in downstream tasks. Here, we will compare the vocabulary before and after filtering, analyzing the changes that occurred.
From Table it can be seen that before processing, the frequency of high-frequency words is higher, indicating higher redundancy in the annotations. After processing, the frequency of high-frequency words decreases, while the frequency low-frequency words slightly increases, suggesting that the overall frequency distribution of the word list becomes more balanced. This also be observed from the range and standard deviation shown in Table where the range and standard deviation of word list frequencies in the processed data smaller.
18

---

Published as a conference paper at ICLR 2025
Table 10: Comparison of the range and standard variance before and after processing.
Indicator
Original Annotation
Processed Data
Range of Word Frequencies
0.4726 0. 1919
0.4431 0.1824
Standard Deviation of Word Frequencies
F ABLATION ANALYSIS ABOUT SELF-BLEU
Regarding the self-BLEU threshold of 0.15 adopted in the benchmark setup, we will discuss some examples here.
Table 11: Some examples with self-BLEU values between 0.15-0.17, where words highlighted in red or in orange indicate higher semantic repetition. It can be observed that while the overall semantics continuously change, certain parts of the meaning remain relatively similar.
Timesangg
Captiees
Timetanpe
Captines
577-1070
13.8-25525
12
14.8-15:22
Ra
26266-8111
15.2322302
29 73-32-88
H[
4001 46.82
49.0999449
4682255.31
ishual
39.9943.111
Tao womee (2
5533:6542
kraa bekeooaa dukke staling
1621-93.56
46 46.99-82.5
8336-13.111
Aeeiilee apd
the pirze oost boliie
000-0065
Twoler in back ciodio di sding
MED E B3 1.E A 14:00:0---9 91
000-6.94
Alee ln Macs proseiing
Seng 52 FRRL FF 18
12.29-16233
6
16.7419911
1NR A 7- 8. B
14.88-23463
lwp coor values dans aviibiol warh www.rraregggg Sn
E. F3 96 1358 02
23.34-38
31.27-0334
SE6656 1.5 (14T 6 ( ARR: (1 C 0111
bured vakile on playce badeeh
34.91445055
ESPES M1211 1:4
caw yalkoo modder free calliin
13 A TI
51.3667705
The addaiiiiiaa a skile joiag ducte huddedul lar heay
1F 19 BulfeEShii 15 7
61.85-75.
Alyy quaita auugardy and apuigelaily wille prosedda of
3 BLSREE A 18-66 16-00 M FF 5 (8 86:088
It's worth noting that for samples far from the 0.15 threshold, such as those with self- values of 0.8 or 0.01, these are extreme cases. Obviously, samples with high self-BLEU values have lower textual diversity, while samples with low self-BLEU values have higher textual diversity. Therefore, to discuss the threshold setting, we will only observe the diversity differences in samples with self- BLEU values around 0. 15, as shown in the Table I and Table T2
From Table T1 we can observe that in samples where the self-BLEU value is slightly above 0.15, there is almost always a certain recurring phrase that appears in all sentences, along with some repeated phrases occurring in most sentences, such as "4 ifk" (singing on stage) or "FT Hi 187"" (greeting) or "ZE > Hi" (in front of the camera), or there are sentences that are semantically almost identical, like (a group of basketball players playing basketball), etc. From Table 12 we can see that repeated phrases only appear in part of the sentences, for example, "К" (playing the piano) and "3f 2i 25 E fl (1
5 A" (a man wearing a suit and glasses) appear in 5 out of 7 instances in our given samples, and there is almost no additional repetition in content. In most samples, repeated phrases like "# प # MAA" (man in suit), " (chatting), or "i i (talking) only appear in a few sentences. Looking at the overall description changes, we can infer that setting the self-BLEU to 0.15 can effectively prevent certain phrases from appearing too frequently in the overall annotation, thereby ensuring the diversity of the annotated text.
G CHINESE CHARACTERISTICS
For instance, admiring the full moon during the Mid-Autumn Festival is an important activity in traditional Chinese culture. The Mid-Autumn Festival, also known as the Mooncake Festival, is celebrated on the 15th day of the eighth month of the lunar calendar. On this day, people gather to
19

---

Published as a conference paper at ICLR 2025
Table 12: Some examples with self-BLEU values between 0.13-0.15, where words highlighted in the same color indicate higher semantic repetition. It can be observed that even the most frequently occurring words appear in only part of the sentences, and there an increasing diversity of repeated words appearing in these sentences.
Timestompp
Capticns
Timeslampe
Captione
0.0014.72
0.00-3335
14.72-20.40
The
5F 38 HI A 1.0 (HO01
1355-15355
10 41 99 6t 21/ R handoora 19 Sie Mt S0 HV 16 2 12:5
them
St FE2 A 1. (::0: ERl: 01
29.48 58,69
The
paininn
roe the greedd and speaking
15.59 20,07
in white lee cluliig spokkng
38,0951449
11 19M F 2.1 1t B-190211
The sea brng ca prouad manaaae caciea
26.00-30.06
of WF EL & (.80 30------ 0,75 ht
vansse in the destang testiis these wakdecs HR 68 99 12 rear woney nectier ieant them quaking
Ss Wa 95 Hii 12:83 E:11 TRII
51,49-6120 26
The aanting run poining an man ortin the groad and quacing
30.0035.72
61.2600
1E ff 7
in ehuck
35.72-44.4
AE-TERE 98-4--5 5. sarrea wanding theee fabling SC
leg hands
69.49-6488
422 (999
44.24-46.62
94,87-97.14
The
ALAEEN I E. R 62 FF
46.02-55.19
HEW FE 9-th it 1t
97.14-10166
I2 T- (1 11% 2::8 It 10000
piak Takin iesenling goge
53.19-58.82
The wuser luzah beg to
3t-- 1 44 41 E3 cia T R.
Maa chelins
0.32-15.49
hadd saa slaras Mark pia
3.46.7.85
A rean in and glaves dasiig on the will
S2 to 56 Bi 1944 A A25 I:
15.49-66117
van in snting meve on plass with green clrc
795-13.04
38 2. 51 Rilik 19.99 11,0 1-001-1116 1e
reax to
planes raacten acorral childers bur dow
36.17.55.55
13,04
burking
& 148 E2 avinn
55.52-60.41
12:01
21.23:24.38
11 1636 fi
69:41-83:70
REFF 1E17--111 II clender hands
74.38-% 94
W SS A-80- ".ll 22 1:9h W 3801
loarsin
11 of (eik 871667799 (K M 56-00
Ie St Hi En Bl 31332117 A I: 10
83.29.9667
saring urrigt in fooe the piarn playing
36,94.39 39.89
child wewiing catoon jucket and mack is bying in drwwiig
0667-117.17
- 1--9 5--22 17 1978 W/B
sae a playing the plane with a baiona expoovics. booking aboui
35 (5
40.59-52.22
Chaldran ludding bruiso gadlec awoad lae tas in aoak and glaves io wakh his dane
2 EE 18---80 # (3 1/ 8R 40 1299999 A 1:
eat mooncakes and admire the full moon, expressing their wishes for family reunion. This cultural practice frequently appears in Chinese videos and annotations but is seldom seen in English datasets. For example, in the Youku Dense Caption dataset, there are videos describing the process of making mooncakes and of people gathering to cat them.
Similarly, the tradition of paying New Year visits is an integral part of Chinese culture. The Chinese New Year is the most important traditional festival in China, marking the beginning of the lunar new year. During this period, paying visits to family and friends to offer blessings and express hopes for prosperous new year is significant activity. This custom holds prominent place in Chinese datasets, with specific practices of New Year visits and celebrations appearing in the Youku Dense Caption dataset. In contrast, such details are often overlooked, simplified, or nearly absent from English datasets, where similar New Year video content is rarely found.
Wrapping Zongzi on the Dragon Boat Festival is also one of the important traditional customs in China. This activity is usually accompanied by family reunion and ancestor worship, which has profound cultural significance and historical origins. For example, the sample of "1 "1108754691" in Figure describes the scene of family gathering for dinner while the elders are cutting Zongzi and preparing to eat Zongzi. Similarly, the Chinese art cross-talk is unique in the Chinese context, with no equivalent in English. These cultural differences manifest not only in the distribution of textual annotations but also in video content.
H MORE DETAILS
Additionally, we examined the issue of average annotation length, where a text annotation length greater than 10 is our minimum requirement. We present a comparison of annotation lengths from some existing manually annotated video caption datasets. As the first Chinese dense caption dataset, Youku Dense Caption not only has the highest number of annotations among manually annotated datasets but also boasts one of the longest average sentence lengths. Other video caption datasets with longer average sentence lengths, such as HD-VILA- 100M with an average sentence length of 32.5 words, achieve this by using Automatic Speech Recognition (ASR) for annotation. It is worth noting that in most cases, ASR cannot accurately describe the visual content and may include a large number of colloquial and irrelevant words.
20

---

Published as a conference paper at ICLR 2025
Table 13: Comparison of Annotation Quantity and Length with Other Manually Labeled Datasets.
Dataset
Total Annotations
Average Annotation Length
MSR- VTT LSMDC
10k
9.3
118k
7.0
YouCook2
14k
8.8
How2
80k
20.0
ANet caption
100k
13.5
Youku Dense Caption
31lk
17.9
MORE CASES IN YOUKU DENSE CAPTION
Video 1089767006
Timestamps
DCH 3-- BL
B Jift F 1-til ar12 approscing nand's
BE A 16 decrcr cakking putlent craditio.
de 1:33 11 Sr faal
11--- A. H9 Hist Namas pataor
Anzotations
Niluadshy the is eegicll
Video
1456583050
Timestapps
1 2 R1 BER
Will Mo
Ansotations
Therch wutido vardaesee
FF
St 80
Then
E
beck slewe tarracking
Mack palling
machines
Thank
vose ralking
Video 1454553050
Timestamps
23.35-29.0
29
36 36.3300
40,69-90777
50.22-54.33
$4.23-22.93
12
Amsolainos
M022 4::0 ( 20
10------------- 26 dodened openiiing devico witt
A to coneii werting her pactaging with
* misolls Gelhcarne.
sromaa dhectder and esant haies
ail dionings laby
madiin with
oppazting the
Vileo ID 1477946140
Timestamps
0.00-6.93
6.3-10.24
10.24-4225
2422536771
42.0 12.01-46.30
46.30-61.93
man pallad plants
kkylooul and pad.
21

---

Published as a conference paper at ICLR 2025
Video ID 1475367533
Timestapps
0.00-4.00
4.00-13.75
13.73 19.86
19.86 30.40
30.40 58.52
63.99 $6.79
10 ti AUIHF EUA 7 95
1O PS fh Add
0: BRR
Ammotation
BEAR # macezall
98 8
produca
muddie big visao with a visuo an
ofe
trach
procoandie the suidd.
ofher haingg prefociinal
bi mouth fot of Begrance
Video iD
1236057735
Timestamps
0.00 1.00
5.8 10.41
22.71-3244
22.42-31.2
50,03 51,59
56.97 62.96
SESSEEEEELL Mi FW----225 1113
X- E3tt
T1 rhspopooere expe
wasur pano
boldnng the swntan on the scrion wat
hos abeet her.
tacincnddile Dace.
22


Reviews:
[{"review_id": "BBlvTe5lvU", "reviewer": "Reviewer_C5mS", "summary": "This paper introduces Youku Dense Caption, a large-scale Chinese dense video captioning dataset. Dataset addresses the scarcity of high-quality Chinese video captioning resources, containing 31,466 short videos with 311,921 Chinese captions. A strategies is proposed to improve benchmark quality by filtering out redundant or low-quality annotations. The authors establish several benchmarks for Chinese video-language tasks and conduct extensive experiments demonstrating the dataset's utility and potential for research. They also discuss challenges related to the linguistic and cultural differences between Chinese and English video data.", "questions": "see weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. A Chinese video captioning dataset is proposed to fill the research gap in the Chinese community for video captioning data.\n2. A embedding-based similarity and a Non-Maximum Suppression method is used to set up a Chinese PRVR benchmark that effectively reduces annotation redundancy.\n3. The work reduces redundancy in video captioning and grounding by filtering out videos with high self-BLEU scores and minimal scene changes,  which is measured through color histogram correlation, ensuring a diverse and representative dataset.", "weaknesses": "1. The statement in the section “Chinese Characteristics” seems unclear. The English translation of the so-called “fine-grained Chinese captions” could also serve as fine-grained English captions in an English-language context. For me, only the localized data part is valuable, as it highlights a major difference between Chinese and English video captions. Adding more data and statistics to support this distinction would strengthen the paper.\n2. In the experiment, when translated back to Chinese. It's kind of blur that which attributes of Chinese dataset lead to the poor performance, the analysis failed to state clearly about the language differences between Chinese and English.\n3. In ablation study, mixing of different datasets only strike a balance between different tasks but failed to achieve idealized performance across different tasks. And the best performance comes from larger data scale rather than data distribution and video-caption pair. \n4. Overall, the dataset serve as a valuable data source for Chinese community in video caption domain, but the value and key attributes of the dataset remain unclear and is not fully proved by the experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8j87SYHKWG", "reviewer": "Reviewer_QnCD", "summary": "The paper presents Youku Dense Caption, the largest publicly available dataset for Chinese dense video captioning, comprising 31,466 videos annotated with 311,921 captions. Collected from the Youku video platform, this dataset addresses the lack of high-quality Chinese video captioning datasets and promotes advancements in Chinese multi-modal research. It provides benchmarks for key tasks such as retrieval, grounding, and generation, and extensive experiments demonstrate its effectiveness on state-of-the-art multi-modal models. The dataset’s scale and quality make it a valuable resource for future research in video-language understanding.", "questions": "1. Vocabulary Statistics and Comparison: Can the authors provide vocabulary statistics and a comparison with other datasets? The captions appear to contain a high degree of repetition.\n2. Threshold for Average Self-BLEU: Why was the threshold for Average Self-BLEU set to 0.15?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Youku Dense Caption contains 31,466 short videos annotated with 311,921 captions, making it the largest dataset for fine-grained Chinese video descriptions.It addresses the scarcity of high-quality Chinese dense video captioning datasets, promoting advancements in Chinese multi-modal models and video-language research.The dataset establishes benchmarks for key video-language tasks such as retrieval, grounding, and generation, with extensive experiments demonstrating the utility of the dataset on state-of-the-art multi-modal models.", "weaknesses": "1. Lack of Caption Diversity and Detail: In Figure 1, the captions for different segments of Video ID 1070344446 show high similarity with little distinction, lacking sufficient variability. Additionally, the descriptions are relatively simple and do not provide background information about the visual content.\n2. Potential Hallucination in Captions: In the D. Implementation Details of Baselines section, the authors mention that they convert videos to 320p resolution and remove the audio component. However, in Figure 1, the second frame of Video ID 1192027222 shows the caption: “The old lady boasts that young women who work hard at chopping can do it.” It is difficult to determine solely from the visual content that the old lady is boasting, raising concerns about the potential for hallucinated captions, especially for those tied to audio-related information.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bGZPXTf49G", "reviewer": "Reviewer_Va4c", "summary": "This paper details a novel Chinese language based dense video caption dataset.", "questions": "- Has the author consider to release the English caption of the proposed dataset? In my opinion, this will broaden the impact of this dataset and benefit more downstream tasks. Please confirm if this is a a planned future work, as well as to discuss the plan for creating high-quality English translation of the captions. Further analysis on the English translation with open-source tool in Fig 1, it is clear that it requires professional proofreading or checked by someone (crowdsourcing?) who are fluent in both Chinese and English language.\n\n- This paper should provide addition details related to the annotation progress. Please provide detilas on: \n1. The total number of human annotators involved.\n2. The qualifications or expertise of the annotators (e.g., native Chinese speakers, etc.) and how are them recruited. \n3. The cost and time spent on annotation per video. \n4. The total number of annotations per annotator and the overall duration. \n5. Any quality control measures sued during the annotation process.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ This work collected a new Chinese based dense video captioning dataset (named Youku Dense Caption). The dataset has several benchmarks for Chinese video-language tasks, including retrieval, grounding, and generation tasks. \n+ This allow developer and researcher to train multmodal foundation model with a fair benchmark. \n+ This submission has validate the impact of large scale dataset on existing multimodal model. Providing empirical evidence of the advantage of rich data in the context of Chinese language.", "weaknesses": "- The proposed dataset's video are evenly sampled from the Youku-mPLUG dataset based on dedicated (sub)categories. So the assumption is that the licensing should not be an issues. To properly handle the copyright concerns, please details the licensing terms for the Youku-mPLUG dataset, and discuss the coverage of usage right, redistribution policies, and any restriction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3zMTEGYwzl", "reviewer": "Reviewer_Qgjf", "summary": "The paper introduces Youku Dense Caption dataset, which is the largest publicly available Chinese dense video captioning dataset for now. The dataset is annotated by human to guarantee the quality of the dataset.\nBuilding upon the proposed dataset, the paper also establishes benchmarks for video-language tasks.\nThe experiments demonstrate that existing state-of-the-art multi-modal models can benefit from this dataset.", "questions": "- How is a long video segmented into multiple clips? What is the splitting criteria?\n\n- Could you show more data samples in Youku Dense Caption dataset? It is helpful for checking the diversity, visual quality, and annotation quality  of the proposed dataset?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper focuses on the topic of Chinese dense video captioning dataset, which is an interesting and under-explored research area.", "weaknesses": "- Doubtful value of the proposed dataset\n    - First, while the proposed dataset is claimed to be a dense video captioning dataset, the collection pipeline is similar to regular video captioning datasets, like HD-VILA-100M or Panda-70M, where a long video is first segmented into multiple clips and each clip is annotated with a caption. Could the authors provide more differences between the dataset collection pipelines of your dataset and a regular video-text dataset?\n    - Second, it is unconvincing to state that \"Chinese and English have significant linguistic differences, so a Chinese dataset is needed\". I appreciate the authors show the errors of translation in line 211 and Section 3.2.2. However, I use ChatGPT to translate the provided samples and it can produce correct results in most of the cases. Take the leftmost sample in Figure 3 as example, I got this: \"A group of motorcyclists is resting by the roadside, chatting.\" from ChatGPT, which is totally correct.\n\n- Lack of necessary experiments: to evaluate the value of the proposed dataset, the authors need to train a model on different datasets and show that the one trained on the proposed dataset is more robust than the others. Such experiment should be conducted on different tasks, such as dense video generation, partially relevant video retrieval. However, none of this experiment is presented.\n\n- It has been shown that long and detailed prompts are beneficial to various tasks, such as video generation. However, the caption annotations are short and less detailed, limiting the value of the dataset.\n\n- For the scene change detection algorithm mentioned in lines 345~364, TransNet-v2 should be more robust than the adopted pixel-based algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)