SlowGuess/NeurIPS_2023_OCR huggingface dataset:
While multimodal models can unlock creative applications that can benefit humanity, they can also enable harmful applications. These include surveillance, especially when people are recorded and the recordings are used without their consent, or generation of harmful content, such as pornographic material. A particularly sensitive topic in this space is disinformation. When model outputs achieve realistic quality, they can be used to create convincing fake content (i.e., deepfakes). These can be exploited to spread fake news, defame individuals or portray false situations. To mitigate these harms, watermarks can be applied to every generated video [66] such that it is possible to to identify whether any given video is generated by a particular model.
Due to the impacts and limitations described above, we remark that STORYBENCH aims to measure progress in text-to-video research. For the same reasons, we do not release our baselines to the public. By no means should our data be extended for use in sensitive domains, but rather for creative goals. We believe that generative technologies like the type of text-to-video models that can be evaluated in STORYBENCH can become useful tools to enhance human productivity and creativity.
The collection of our datasets has been enabled by the careful work of several participants. Due to privacy concerns, we did not include the estimated hourly wage paid to them or the total amount spent on participant compensation. We feel that individuals' hourly wage or compensation is personal information and we cannot disclose this under privacy law. However, this work was carried out by paid contractors, and we can confirm that they received their standard contracted wage, which is above the living wage in their country of employment.
STORYBENCH was created to encourage reproducible progress in text-to-video modeling. Existing video captioning datasets consist of a single sentence describing the salient events that happen throughout the entire video. Existing dense video captioning datasets, instead, are either domain-specific (e.g., instructional video) or contain captions that lack enough information to generate a video. With our annotation protocol, we describe each action separately and also map it to a precise timestamp interval, allowing us to evaluate the ability of text-to-video models to generate arbitrarily long stories. Our task of continuous story visualization is closely related to the existing one of story
visualization, which was, however, limited to generate a single key-frame per caption, rather than a continuous video. With the release of STORYBENCH, we aim to establish a framework for reliable evaluation of forthcoming generative video technologies.
STORYBENCH was collected by Google Research.
Google Research funded the creation of STORYBENCH.
We provide 8,900 annotations of stories across six splits. Each story contains an object serialized in JSON with the following fields: sentence Parts, start-times, end-times, original(combined)sentence, clip_start_time, clip_end_time, story_number, background_description, dataset_name, video_name, vidln_id, question_info, num Actors_in Video, segment categories. We provide a description of each field in the README file of our code online. In addition to our annotations, an instance of the dataset requires the corresponding video file from existing datasets.
The DiDeMo-CSV dev split has 744/744 videos/stories, and the test split has 655/655 videos/stories. The Oops-CSV dev split has 979/1578 videos/stories, and the test split has 979/1578 videos/stories. The UVO-CSV dev split has 1019/1665 videos/stories, and the test split has 1565/2613 videos/stories.
STORYBENCH consists of annotations from a subset of dev and test videos from DiDeMo, Oops, and UVO. The annotated videos were selected based on a few criteria: (i) public availability as of February 22, 2023; (ii) lack of inappropriate content; (iii) annotation quality insurance; (iv) preprocessing criteria (e.g., by removing videos whose first action last less than 1.5s).
We provide raw annotations and corresponding video filenames (text). In addition, we also release the features used to compute our set of automatic metrics for the ground-truth videos.
The goal of the dataset is not to classify any given instance. However, we enrich the annotation of each action to easily analyze failure modes by collecting 35 labels across 6 categories (camera movements, foreground entities, foreground actions, background actions, foreground interactions, foreground transitions). We provide the full list of labels in the main body of the paper.
Q12 Are there recommended data splits (e.g., training, development/validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
Yes. We collect annotations for the existing dev and test splits. We thus recommend using the original training/dev/test splits to avoid any leakage.
Some videos have multiple stories, which correspond to different instances in our datasets. For data collected in VidLN [20], these correspond to descriptions centered around different actors.
Our benchmark and the datasets we collected rely on existing video datasets (DiDeMo, Oops, and UVO). We do not provide archival versions of the complete datasets, but the corresponding video resources are publicly available for download from their official websites.
Our collected annotations have been verified by humans not to contain inappropriate content. Moreover, our annotators flagged videos that contained sensitive data, which were then all discarded. While we did make an attempt to remove inappropriate content, we cannot exclude that a small number of inappropriate samples might have gone unnoticed.
Q17 Does the dataset relate to people? If not, you may skip the remaining questions in this section. Several of our descriptions and corresponding videos are about people. All of the datasets have been verified for sensitive content, and several instances do not include people.
We do not explicitly collect annotations for any subpopulation. However, it may still be possible to deduce this information from the videos and/or the written descriptions.
Yes, it may be possible to identify people from the videos corresponding to our annotations.
Yes, our data might be considered sensitive. For instance, the associated videos reveal racial or ethnic origins of people shown in them. However, we note that we removed any videos that were found inappropriate by our annotators.
We call for responsible usage of our datasets for research purposes only given the potential of text-guided video generation technologies to affect users.
Q22 How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly
inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If data was reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.
We collected human annotations from existing, publicly available video datasets. During the collection campaign, our annotators directly looked at the raw videos. A random sample of the annotations were verified by other humans to ensure high-quality standards.
We collected human annotations through web user interfaces that we developed. They were validated by manual inspection by us and managers from the company we hired to collect human annotations.
We annotated all the videos from the evaluation sets of existing datasets that were still available online at the time of our data collection. We also discarded any videos that were found inappropriate.
We hired a third-party company to collect human annotations from contractors, who received their standard contracted wage, which is above the living wage in their country of employment. The first and last author were also closely involved during the data collection to ensure that the instructions were clear and resolve any doubts raised by the crowdworkers.
Our video annotations were collected between December 2022 and March 2023, but the corresponding videos were previously collected by other authors.
No institutional review board conducted any ethical review process since we do not modify the original videos, and the datasets providing the videos are publicly available and have previously been published in peer-reviewed journals and conferences.
Yes, people may appear in our annotations as well as in the corresponding videos.
We collected annotations from crowdworkers, not from the individuals shown in the original videos.
Individuals were not notified about our data collection, which involved describing their actions in publicly released videos.
Q31 Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
We collect annotations from existing, publicly available video datasets. We do not, however, annotate videos that were no longer available online at the time our annotation campaign was conducted, to adhere with the users' intent to remove their content online.
Q32 If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).
Users can check whether any of their videos is used in our datasets from the corresponding URLs. If users wish to remove their videos after finding them sensitive, they can contact the hosting party
and request to delete the content from the underlying website. Users can also contact us to request removal of the instances in our datasets corresponding to their videos.
Q33 Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.
The goal of our datasets is to encourage research towards generative models that can assist and boost artists in generating novel content. However, the resulting technologies could be used to create misinformation online, such as through deepfakes. Yet, we believe that our datasets are the first of their kind to study the problem of generating videos from captions that vary over time. Hence, considering both limitations and opportunities offered by our data, we authorize the dataset for purely academic endeavors.
Preprocessing, Cleaning, and/or Labeling
Yes, we ask human annotators to select which of 34 labels are related to any video segment and captions. We remove any instances (i) whose first action lasts less than 1.5s, or (ii) have a timestamp gap longer than 0.5s between any two consecutive actions.
Q36 Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
No, we do not save the raw data due to data retention policies in our organization.
Q37 Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.
Yes, we release our preprocessing scripts on GitHub.
Our datasets have not been used for other tasks yet. However, the underlying videos have been used for their original tasks, such as temporal localization with DiDeMo, studying unintentional human action with Oops, and dense, open-world segmentation with UVO. Moreover, VidLN annotations have been used for the tasks of video narrative grounding and video question answering.
Q40 Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.
We encourage the community to measure progress in our benchmark and datasets at the URL https://paperswithcode.com/dataset/storybench.
Our data can be used for the dual task of describing videos over time. In addition, our data could be used to develop automatic evaluation metrics that better align with human preferences. We also believe that the richness of our data will encourage future work to create new, exciting tasks.
Our annotations describe existing video datasets that might not contain a fair distribution of individuals or groups. For our task, this means that models might be able to generate videos that are biased
towards the populations represented in the training data. We encourage future work to extend our efforts towards creating training and evaluation datasets that specifically aim to increase fairness and reduce biases (e.g., correlation between gender, race and jobs) of generative text-to-video models.
Q43 Are there tasks for which the dataset should not be used? If so, please provide a description. Under no circumstances should any models developed for our benchmark be used to create deepfakes or any other form of disinformation or harm, including military and surveillance tasks. As it stands, our datasets should solely be used for research purposes.
Q44 Any other comments?
Q45 Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.
Yes, the data will be publicly released.
Q46 How will the dataset be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?
The data will be available on GitHub.
Q47 When will the dataset be distributed?
From September 2023 and onward.
No, our collected annotations are released under a CC-BY-4.0 license. Third-party data are also released publicly.
Google Research will support and maintain the STORYBENCH annotations on GitHub. The original videos are supported by the corresponding dataset creators or services.
We can be contacted either via email or through 'pull requests' on the STORYBENCH GitHub page.
There is no erratum for our first release. Errata will be documented as future releases on GitHub.
No, we do not plan on updating the data. However, we will update the data should there be any errors or requests for deleting specific instances. The updated data will be shared as a 'release' on GitHub.
We do not collect any metadata related to people in creating STORYBENCH. However, we note that our datasets consist of text annotations of existing video datasets. Should people request for their videos to be deleted from the original datasets, we invite them and users to contact us to ensure that the corresponding annotations are removed from STORYBENCH.
Yes, we will distribute all versions of STORYBENCH as 'releases' on GitHub.
There is no plan to support and verify third-party contributions that aim at extending the datasets in STORYBENCH as our annotations correspond to standard evaluation splits of existing video datasets. However, we will update the data should there be any errors or requests for deleting specific instances. Dataset versions will be maintained as GitHub releases.
D Data Preparation Details
In this section, we provide further details on the preparation pipeline used for the evaluation (dev and test) data in STORYBENCH. It consists of the following steps: collection, preprocessing, and rating.
D.1 Data Collection
First, we use an online interface (see Figure 5) to collect stories for each video. Stories consist of multiple sentences, each describing an action, and the corresponding timestamps in the video.
Oops-CSV and UVO-CSV. For VidLN data (UVO and Oops), we provide the original VidLN caption, as well as reference split captions provided from our automatic pipeline  $(c.f.$  Section 4). In this stage, annotators were instructed to 'split the long sentence into shorter sentences, each describing actions that happen one after the other' and to 'add time stamps for when (the action of) each sentence starts and ends.' Moreover, our annotators are asked to click two checkboxes, whenever applicable: 'Multiple stories,' used to indicate whether the video shown in the user interface actually consists of multiple shorter clips (this is common in Oops, as the data consists of fail video compilations); and 'Unimportant actor,' used to indicate whether the original caption describes the events in the video from the perspective of an entity that does not play a salient role in the video (e.g., a person in the background). Finally, we perform a second stage of annotations where we provide annotators with the stories from the first stage, and ask them to 'continue the sentence that describes the actor's action in a natural manner by adding a concise context description of relevant actions of other actors.' This second round of annotations was required as VidLN describes the actions of a given entity (actor) throughout a video, which does not often capture the dynamics of the corresponding video segments. Our annotators narrate 2,446 and 2,779 videos from the dev sets of Oops and UVO, respectively.
DiDeMo-CSV. For DiDeMo, we do not possess any descriptions of the video. Instead, the dataset provides detailed text queries (e.g., containing camera movement, temporal transition indicators, and activities) that are used to localize events in the video. We provide those queries as a reference to our annotators, and ask them to 'add a description of the background,' 'specify the number of important actors in the video,' 'refine the sentences to create a coherent story,' and 'add timestamps for when (the action of) each sentence starts and ends.' Following the original DiDeMo protocol, each video is trimmed to a maximum of 30 seconds. While the original dev and test sets contained 1,065 and 1,004 videos, respectively, only 843 and 797 videos were still publicly available as of February 22, 2023.
Figure 5: Example of our video annotation interface.
Sentence part reference	Sentence part	Start time	End time
Another woman wearing a white top is standing on the right.	Another woman wearing a white top is standing on the right.	0.613003	1.074029
The woman is taking the crab from the first woman.	The woman is taking the crab from the first woman.	1.175798	3.462071
The woman starts screaming as the crab bites her.	The woman starts screaming as the crab bites her.	3.563038	10.178444
Multiple stories Unimportant actor
Segment caption: A brown-white dog is standing on the right side of a canal and trying to catch a white ball from the canal with its paw and moving its paw in the water again and again.
Camera movements
Figure 6: Example of our diagnostic labels collection interface.
static shot
Text □	Video □
pan
Text □	Video □
tilt
Text □	Video □
zoom
Text □	Video □
tracking shot
Text □	Video □
over-the-shoulder-shot
Text □	Video □
aerial shot
Text □	Video □
point-of-view-shot
Text □	Video □
-Entities
people
Text □	Video □
animals
Text □	Video □
food/drinks
Text □	Video □
tools
Text □	Video □
containers
Text □	Video □
Segment caption: The brown-white dog goes to the other side and picks the ball with its mouth.
- Camera movements
static shot
Text □	Video □
pan	Text □ Video □
tilt	Text □ Video □
zoom	Text □ Video □
tracking shot	Text □ Video □
over-the-shoulder-shot	Text □ Video □
aerial shot	Text □ Video □
point-of-view-shot	Text □ Video □
-Entities
peoples
Text Video
animals
Text Video
food/drinks
Text Video
tools
Text Video
containers
Text Video
Segment caption: The brown-white dog jumps to the right and drops the ball in the canal.
static shot—
Text □ Video □
pan
Text □ Video □
tilt
Text □ Video □
zoom
Text □ Video □
tracking shot
Text □ Video □
over-the-shoulder-shot
Text □ Video □
aerial shot
Text □ Video □
point-of-view-shot
Text □ Video □
Segment caption: The brown-white dog is looking towards the ball.
—Entities 
people
Text □	Video □
animals
Text □	Video □
food/drinks
Text □	Video □
tools
Text □	Video □
containers
Text □	Video □
In both cases, we provide additional details for both of these tasks to the annotators, as well as examples and corner cases to clearly communicate the desired annotations (available on GitHub). During this collection process, any video flagged by the annotators to contain inappropriate content was removed. Finally, a random sample of our annotations were verified by expert annotators identified by the third party company responsible for human annotations in this project.
Figure 7: Distribution of collected labels per category in our dev samples.
Figure 8: Distribution of collected labels per category in our test samples.
Diagnostic labels. After collecting story annotations for our videos, we enrich them with labels to help analyze the performance of forthcoming text-to-video models along different axes. With the help of artists that have been using generative AI technologies, we define 34 labels across six categories (c.f. Section 3). For each video segment, we then ask our annotators to tick two checkboxes per label: 'Text' if the label is mentioned in the segment caption; and 'Video' if the label is shown in the video. Figure 6 shows an example of the UI used in this process, and we release our full set of instructions online. Figures 7 and 8 show the distribution of labels in our preprocessed (i.e., final) data.
Human annotation framework. We assess the high quality of our annotations as follows. First, annotators were only moved to the final data annotation process after having successfully completed a training stage. Second, we asked annotators' managers to verify the quality (i.e., descriptions and timestamps match the content of the videos) of the final data by manually checking  $25\%$  of the data samples. Here, they found  $97\%$  of the samples to accurately reflect the narratives of the videos.
D.2 Data Preprocessing
Given the above collected annotations, we perform the following two preprocessing steps. First, we only keep stories whose first action is at least 1.5s long; as we use a video of 0.5s to condition text-to-video generation for the task of story continuation. Second, we remove any story in which two subsequent actions have more than 0.5s gap.
Figures 9 and 10 shows our final data distributions. For DiDeMo-CSV, the dev split has 744/744 videos/stories, while the test split has 655/655 videos/stories. For Oops-CSV, the dev split has 979/1578 videos/stories, while the test split has 979/1578 videos/stories. For UVO-CSV, the dev split has 1019/1665 videos/stories, while the test split has 1565/2613 videos/stories.
The preprocessed data is then adapted for each of our evaluation tasks detailed in Section 3: action execution, story continuation, and story generation. Finally, to evaluate our baselines, the original videos are downsampled to 8 frames per second (fps) using the 'FFmpeg' open-source software.
Figures 12 to 14 show examples of the resulting data.
D.3 Human Evaluation
Human evaluation is the preferred way to assess the capabilities of generative models. We perform side-by-side comparisons between two models, and ask human raters to choose the one (if any) that performs better according to the five criteria that we defined in Section 3. Figure 15 shows an example of the user interface developed for human evaluation.
D.4 Automatic Evaluation
Section 3.4 introduces our automatic evaluation metrics. Here, we provide our intuition of how we expect them to relate to our human evaluation metrics. FID would measure "visual quality" since it compares the distribution of ground-truth frames with that of generated frames. FVD would measure "entity consistency" and "action realism" since it compares the distribution of ground-truth videos with that of generated videos. SIM would measure "visual quality", "entity consistency", "background consistency", and "text adherence" as it compares ground-truth and generated frames one-to-one. VTM would measure "text adherence" as it compares generated videos to their prompts. PQA would measure "visual quality" and "action realism" as it was trained to predict the average human subjective perception of a video.
D.5 Robustness of Automatic Pipeline