SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Answer: This evaluation dataset is created by Yuanxin Liu, Lei Li, Shuhuai Ren, Rundong Gao, Shicheng Li, Sishuo Chen and Xu Sun (Peking University) and Lu Hou (Huawei Noah's Ark Lab).
Who funded the creation of the dataset?
Answer: Peking University and Huawei Technologies Co., Ltd.
I.2 Composition
What do the instances that comprise the dataset represent? (e.g., documents, photos, people, countries)
Answer: Our data is provided in JSON format. Each data instance consists of (1) a text prompt, (2) the categorization labels under three aspects, (3) the source of the prompt (collected from MSR-VTT and WebVid or manually written by us), (4) ID and (5) URL link of the reference video.
Answer: In total, we collect 619 instances. The data distribution over different categories can be found in Figure 3 and Figure 4 of the main paper.
Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set?
Answer: A part of the text prompts in FETV is collected from existing datasets of open-domain text-video pairs (i.e., MSR-VTT and WebVid). When collecting prompts, we ensure that each category in our proposed categorization system has sufficient prompts.
Is there a label or target associated with each instance?
Answer: Each prompt collected from MSR-VTT and WebVid is associated with a reference video, which is provided in the form of an URL link.
Is any information missing from individual instances?
Answer: For the manually written unusual prompts, FETV does not provide the URL link to reference videos.
Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)?
Answer: The text prompts in FETV belong to different categories, which can be seen from the categorization labels.
Are there recommended data splits (e.g., training, development/validation, testing)?
Answer: No. The entire FETV benchmark is intended for evaluation.
Are there any errors, sources of noise, or redundancies in the dataset?
Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)?
Answer: The reference videos are provided as URL links because we do not own the copyright to the videos.
Does the dataset contain data that might be considered confidential?
Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety?
I.3 Collection Process
The data collection process is described in Section 3.2 of the main paper and Appendix G.
I.4 Uses
Has the dataset been used for any tasks already?
Answer: Yes. We have used the FETV benchmark to evaluate three existing text-to-video generation models and diagnose the reliability of automatic video quality and video-text alignment metrics. Please see Section 4 and Section 5 of the main paper for details.
Answer: FETV is mainly designed to evaluate open-domain text-to-video generation models, but it can also be extended to test the reliability of automatic evaluation metrics, by computing their correlation between human judgements (as shown in Section 4).
Is there a repository that links to any or all papers or systems that use the dataset?
Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses?
Answer: In the first stage of our data collection process, some prompts that actually belong to a particular category might be filtered out by the automatic matching rules (introduced in Appendix G), which potentially impacts the language diversity of the prompts.
Are there tasks for which the dataset should not be used?
Answer: FETV mainly consists of sentence-level text prompts, and thus it is not suitable for evaluating video generation based on paragraph-level texts.
I.5 Distribution
Answer: Yes. The benchmark is publicly available on the Internet.
Answer: The benchmark is available on GitHub at https://github.com/1lyx97/FETV or on Huggingface at https://huggingface.co/datasets/lyx97/FETV.
Answer: CC-By 4.0.
Have any third parties imposed IP-based or other restrictions on the data associated with the instances?
Do any export controls or other regulatory restrictions apply to the dataset or to individual instances?
I.6 Maintenance
Answer: The authors will be supporting, hosting, and maintaining the dataset.
Answer: Please contact Yuanxin Liu (liuyuanxin@stu.pku.edu.cn).
Is there an erratum?
Answer: No. We will make announcements if there are any.
Answer: Yes. We will post new update in https://huggingface.co/datasets/1yx97/FETV and https://github.com/llyx97/FETV if there is any.
If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were individuals in question told that their data would be retained for a fixed period of time and then deleted)?
Answer: People may appear in the reference videos. People may contact us to exclude specific data instances if they appear in the reference videos.
Will older versions of the dataset continue to be supported/hosted/maintained?
Answer: Yes. Old versions will also be hosted in https://huggingface.co/datasets/lyx97/FETV and https://github.com/1lyx97/FETV.
Answer: Yes. Others can extend FETV to collect more prompts for evaluation, based on our multi-aspect categorization system (introduced in Section 3.1 of the main paper).