SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Emanuele Bugliarello\*f,  $\tilde{\mathbf{0}},\tilde{\mathbf{c}}$
Hernan Moraldo
Ruben Villegas
Mohammad Babaeizadeh
Mohammad Taghi Saffar
Han Zhang<sup>3</sup> Dumitru Erhan<sup>3</sup>
Vittorio Ferrari
Pieter-Jan Kindermans
Paul Voigtlaender
<sup>c</sup>University of Copenhagen
https://github.com/google/storybench
Figure 1: We propose STORYBENCH, a benchmark for continuous story visualization tasks where a model generates a video given a sequence of text prompts and their duration, as well as a video prompt for two tasks.
Generating video stories from text prompts is a complex task. In addition to having high visual quality, videos need to realistically adhere to a sequence of text prompts whilst being consistent throughout the frames. Creating a benchmark for video generation requires data annotated over time, which contrasts with the single caption used often in video datasets. To fill this gap, we collect comprehensive human annotations on three existing datasets, and introduce StoryBench: a new, challenging multi-task benchmark to reliably evaluate forthcoming text-to-video models. Our benchmark includes three video generation tasks of increasing difficulty: action execution, where the next action must be generated starting from a conditioning video; story continuation, where a sequence of actions must be executed starting from a conditioning video; and story generation, where a video must be generated from only text prompts. We evaluate small yet strong text-to-video baselines, and show the benefits of training on story-like data algorithmically generated from existing video captions. Finally, we establish guidelines for human evaluation of video stories, and reaffirm the need of better automatic metrics for video generation. StoryBench aims at encouraging future research efforts in this exciting new area.
*Work completed during an internship at Google. Correspondence to emanuele@di.ku.dk.
There has been an explosion in the capabilities of generative AI models in the last two years. Recent language models can produce fluent text [e.g., 1-4], audio generation models can synthesize high-fidelity speech and music [5-7, i.a.], and text-to-image models can create photorealistic images [e.g., 8-11], paving the way to real-world applications of AI in art, design and content creation.
Generating high-quality videos of arbitrary duration from text is still, however, an arduous task. Compared to text-to-image synthesis, video generation introduces several distinctive challenges. First, videos should be coherent over time (e.g., objects should be consistent from frame to frame) while reflecting the actions described in text prompts. Second, training text-to-video models is computationally expensive: multiple frames need to be synthesized within the same context to produce smooth transitions over time. And third, video-text datasets [e.g., 12-14] are orders of magnitude smaller than corresponding image-text datasets [15, 16, i.a.].
To enable researchers to reliably measure progress in video generation from text prompts that change over time, we introduce the STORYBENCH benchmark. We create STORYBENCH by collecting dense, rich annotations for the validation and test sets of three existing, general video datasets: DiDeMo [17], Oops [18] and UVO [19], for a total of 6K videos. Our human annotators first describe a video with a sequence of captions, one for each action, forming the story of the video. In this stage, our annotators also temporally segment the full video accordingly, providing timestamps for each caption. This results in unique datasets, with rich annotations of a sequence of actions (i.e., a story) that have clear timestamps. Moreover, we next ask annotators to tag each of the resulting 18K video segments according to 34 labels that span 6 categories, such as camera movements and foreground interactions, allowing researchers and practitioners to easily pinpoint the limitations of their text-to-video models. Finally, given the challenges of obtaining high-quality training data for story visualization, we devise a pipeline to automatically transform existing, grounded video captions [20] into stories, resulting in training data that is very rich in structure, with (i) a sequence of action descriptions, (ii) their timestamps, and even (iii) mouse traces and segmentation masks. We show that this transformation leads to better models, and encourage future work in data curation strategies to improve performance.
Based on these data, we define three tasks of increasing difficulty (see Figure 1) with the goal of synthesizing videos that follow a sequence of text prompts (i.e., a story). In the action execution task, a text-to-video model is simply asked to continue a conditioning video from a single caption (i.e., a short sentence describing a single action). This task is extended in story continuation, where the model must generate a sequence of coherent video segments given a conditioning video and a series of captions. Finally, the story generation task also asks a model to generate a sequence of coherent video segments from a series of captions, but without a conditioning video. Instead, the model is first asked to generate its own conditioning video from a textual description of the expected scene background. For each of the captions, models are also given the expected duration of the corresponding video segment (e.g., 1.5s). This requires models to be controllable, resembling real-world applications of artistic content creation. Overall, we name the overarching challenge continuous story visualization.
We train a small, 345M parameter text-to-video Phenaki [21] model. In STORYBENCH, models can be evaluated in three different setups: zero-shot, where a model is tested after general-purpose pretraining, given no STORYBENCH data; single-task, where a model is separately fine-tuned on each dataset; and multi-task, where a single model is fine-tuned on our three datasets. Due to the challenges in evaluating videos, we conduct a human study by defining guidelines that capture different desired qualities of continuous story visualization. We also report performance across five automatic metrics, but we show they do not align well with human judgment; reasserting the need of better metrics.
Contributions. In this work, we 1) introduce STORYBENCH, the first benchmark for continuous story visualization, to inspire more work on real-world, text-to-video generation of arbitrary duration through a reproducible and comprehensive setup. 2) STORYBENCH contains rich human annotations, consisting of action-by-action descriptions, their timestamps, multimodal grounding via mouse traces, and labels for each segment to easily determine failure modes. STORYBENCH is a truly multifaceted benchmark: it spans three tasks of continuous story visualization, across three different datasets, and three evaluation setups (zero-shot, single- and multi-task). We 3) devise an automatic procedure to transform video captions into a sequence of captions, each describing a single action, and show the effectiveness of using such rich data during model fine-tuning. We 4) establish a compact yet strong baseline, define guidelines for human evaluation, as well as a set of automatic metrics for reproducible comparisons of the next generation of text-to-video models. 5) Our results show the
benefit of training text-to-video models to continue videos (rather than generating them from scratch) on story-like data, but they also suggest discrepancies between human and automated evaluation. We invite the community to report results on STORYBENCH at https://paperswithcode.com/dataset/storybench. We provide data, annotation instructions, human evaluation guidelines, and code for automatic evaluation at https://github.com/google/storybench.
Text-guided generative models for vision. We are currently witnessing tremendous progress in the task of text-to-image generation and editing [8-11, 22-27], fueled by sequence-to-sequence Transformer models [28] and diffusion models [29] trained on massive amounts of image-text data [15]. Research on text-to-video generation has also received attention recently. GODIVA [30] autoregressively generates videos from text using a three-dimensional sparse attention mechanism. N'UWA [31] presents a unified framework for multi-task learning of various generation tasks, including text-to-video. NUWA-Infinity [32] is a generative model that can synthesize arbitrarily-sized images or long-duration videos with an autoregressive over autoregressive generation mechanism. In a similar spirit, N'UWA-XL [33] proposes a diffusion over diffusion approach that allows to generate long videos in parallel through a coarse-to-fine process. CogVideo [34] adds temporal attention modules on top of a frozen text-to-image model to reduce the computational requirements for text-to-video learning. Make-a-Video [35] also starts from a text-to-image model but fine-tunes it while adding pseudo-3D convolution and temporal attention layers. Concurrently, Video Latent Diffusion Models [36] turn pretrained image diffusion models into video generators by fine-tuning them with temporal alignment layers, and Imagen Video [37] generates high definition videos using a cascade of video diffusion models. Ho et al. [38] train space-time factorized U-Net models [39] on images and videos, and propose a sampling method to improve longer video generations. Our baselines are based on Phenaki [21], which can generate arbitrary long videos from a sequence of text prompts.
Story visualization. In this paper, we propose STORYBENCH, a benchmark for the task of generating a video from a sequence of text prompts (i.e., a story), which we refer to as continuous story visualization. In the literature, story visualization [40] is the task of generating a sequence of images to narrate a multi-sentence story (one image per sentence) with a global visual consistency across dynamic scenes and entities. The authors created two artificial datasets from CLEVR [41] and Pororo [42], and proposed a model based on sequential conditional GANs. To improve story visualization, Maharana et al. propose a dual learning framework and a copy mechanism in [43], and leverage grammatical and visual structure as well as commonsense information in [44]. In [45], the authors introduce the DiDeMoSV dataset, and propose to 'retro-fit' a pretrained text-to-image model with task-specific modules to improve on the task of story continuation, resulting in StoryDALL-E. Finally, Rahman et al. [46] extend the synthetic MUGEN dataset [47] for multi-sentence storylines, and propose an autoregressive diffusion-based framework with a visual memory module to capture the entities and background context across the generated frames. Unlike previous work, in STORYBENCH, we focus on generating continuous videos (rather than key-frames) on natural (rather than cartoon or synthetic) data. Moreover, we also use DiDeMo to visualize stories but rather than using the existing temporal queries and automatically matching them to key-frames [45], we ask human annotators to thoroughly describe the story of the videos while manually annotating timestamps for each sentence.
3 StoryBench
Aiming for a comprehensive resource to assess the ability of generative models to visualize stories, we propose STORYBENCH, the first real-world benchmark for text-to-video story generation. Unlike previous work which frames story visualization as the task of generating a single key-frame per text prompt, STORYBENCH evaluates the ability of generative models to synthesize continuous, natural videos from a sequence of text prompts. To do so, we collect rich annotations that provide insights and nuances of any model's capabilities, and easily discover failure modes. STORYBENCH consists of three different datasets, three tasks of increasing difficulty, and three evaluation setups.
Generating videos is a very complex task for state-of-the-art models. Some of the key challenges involve generating videos that (i) have a coherent storyline, (ii) are visually realistic, and (iii) can be controlled according to user intent. STORYBENCH aims at benchmarking these three challenges by (i) defining three tasks with increased difficulty in storyline; (ii) focusing on real-world, natural videos; and (iii) enabling control over the generated content through text descriptions and video duration.
Dataset	#Videos	#Stories (per video)	#Segments (per story)	#Words (per story)	#Labels (per story)	#Actors (per video)
DiDeMo-CSV	1,399	1,399 (1.00)	4,926 (3.52)	80,405 (57.47)	14,463 (10.34)	3,228 (2.31)
Oops-CSV	1,888	3,243 (1.72)	7,198 (2.22)	131,485 (40.54)	30,027 (9.26)	6,556 (3.47)
UVO-CSV	2,917	4,258 (1.46)	6,227 (1.46)	122,542 (28.78)	36,751 (8.63)	7,743 (2.65)
Table 1: Statistics of our collected evaluation datasets. Actors refer to entities with a key role a video.
Category	Labels
Camera Movements	static shot, pan, tilt, zoom, tracking shot, aerial shot, point-of-view shot
Foreground Entities	people, animals, vehicles, food/drinks, containers, tools
Foreground Actions	humans moving, animals moving, objects moving, humans using objects, animals using objects, static actions
Background Actions	animate entities moving, objects moving, animate entities using objects, static actions, dynamic background
Foreground Interactions	dialogues, direct, indirect, object-based
Foreground Transitions	new entities, new objects, entities vanish, objects vanish, entities re-enter, objects re-enter
Table 2: Overview of categories and labels for each video segment to easily detect failure modes.
We annotate three different, real-world, open domain video datasets for continuous story visualization. Table 1 lists high-level statistics of our evaluation data. See App. D for more details and data samples.
DiDeMo-CSV. DiDeMo [17] is a dataset collected for the task of temporal localization, where a model is tasked to identify the video segment corresponding to a given text query (e.g., 'the cat jumps on the mat'). The authors define segments with a resolution of 5 seconds. Due to the nature of the temporal localization task, the original text descriptions are often short and lack essential context information to be used for story visualization. We hence ask human annotators to collect descriptions of the actions happening in a video in order to create a coherent story. The annotators are also tasked to (i) add timestamps for each action, (ii) add a description of the background (to be used for the story generation task), and (iii) specify the number of important actors throughout the video.
Oops-CSV. Oops [18] is a dataset covering unintentional human actions (i.e., fail videos), originally filmed by nonprofessionals, that are diverse in actions, environments, and intentions. This kind of fail videos are extremely interesting for video generation, as they are unpredictable. A failure is typically a surprising, unexpected action, and one that an ordinary video model would not likely generate. To use these videos for the task of continuous story visualization, we start from the rich annotations in VidLN [20], which consist of descriptive captions that are grounded to the videos through mouse traces. First, we design a pipeline to automatically split the original captions into a sequence of actions that keep most of the original words (see Section 4). Second, we ask human annotators to refine these preprocessed captions by (i) fixing any errors in sentence splitting whilst keeping most of the original words (in order to preserve the rich annotations from VidLN), (ii) adding timestamps for each action, and (iii) adding a concise context description. In fact, the VidLN captions describe the actions of a single entity (i.e., actor) throughout the video. Upon inspection, we found that such captions were unsuitable for our task, as they often lacked information that was crucial to synthesize a video similar to the ground-truth. The VidLN data already specified the number of important actors throughout the video, which we update based on stories that were kept after a quality insurance phase.
UVO-CSV. UVO [19] is a dataset originally collected for open-world, class-agnostic object segmentation in videos. Videos in UVO are sourced from Kinetics-400 [48], which contains 10-second 30fps videos labeled with human actions. UVO includes many videos with crowded scenes and complex background motions, taken by both professionals and amateurs. As UVO videos have also been annotated with VidLN data, we follow the same steps as for Oops-CSV to create UVO-CSV.
Segment-level labels for easy diagnostics. In addition to data for visual generation of stories, we collect rich annotations to promptly detect failure modes of text-to-video models in STORYBENCH. Specifically, together with artists and designers that work with generative AI, we defined 34 labels across 6 categories, shown in Table 2. These include: camera movements, to determine the type of shot; foreground entities, to know which entities are shown in the video; foreground and background actions, to know which actions happen in the video; foreground interactions, to know how different entities interact with each other; and foreground transitions, to know how entities evolve throughout
2We only annotated videos from the dev and test sets that were still available online as of February 22, 2023.
3Important actors are entities (e.g., people, animals) that perform actions central to the story of the video.
the video. For each video segment and label, annotators were asked to check two boxes indicating whether a given label was shown in the video and/or in the caption. By doing so, we can study whether models can capture specific aspects of video generation while also knowing if those aspects were mentioned in the text prompt. We provide further details on each category and label in App. D.1.
Human annotation framework. We rely on an in-house platform to annotate our data. For any kind of annotations, human annotators first undergo a training phase. In this phase, they raise any doubts and present corner cases to the first author, who revisits the guidelines with further details and examples. After training, the first author and last author meet with the annotators' managers to ensure the task is clearly understood. The managers then assess which annotators have a high performance for the task, before collecting the final annotations. The annotators' managers, the first author and the last author communicate regularly throughout the annotation process, validating samples and discussing any potential issues (e.g., removing videos with problematic and offensive content).
We define three tasks for continuous story visualization of increasing complexity (shown in Figure 1). We call context the history of all text and/or video prompts available to a model at any generation step. The input, instead, includes a text prompt describing the next video to be generated, and its duration.
Action execution. The task of action execution simply requires a model to generate the next action specified in the input. The context is given by the ground-truth video preceding the generation step. Whenever the first action of a video must be generated, models are conditioned on the first 0.5s of ground-truth video, in order to provide visual context whilst being robust to camera movements.
Story continuation. The task of story continuation requires a model to generate a video from a sequence of inputs and a conditioning video. Analogous to action execution, whenever the first action of a video must be generated, models are conditioned on the first 0.5s of ground-truth video. The context for the following steps also includes all the previous text prompts and synthesized videos.
Story generation. The task of story generation also requires a model to generate a video from a sequence of inputs, but without any ground-truth visual conditioning. Instead, the context for the first generation is a video synthesized by the model from a text description of the background, to guide the space of possible outputs. The following steps are generated analogously to story continuation steps.
Evaluation of generative models for perceptual data, such as images and videos, is critical to validate their performance. In fact, humans possess complex cognitive abilities to interpret visual content, which automatic metrics often fail to capture. Further, we are ultimately interested in the impact and utility of generative models on users, and human evaluation provides such user-centric perspective.
We follow previous work [10, 21, 22, 35] in doing side-by-side evaluations, where human raters are asked to choose the best (or none) of two outputs for the same prompt. We extend the common metrics of visual quality and text adherence to capture crucial aspects of story visualization as follows.
Visual quality: Which video looks better?
Text adherence: Which video better reflects the caption?
Entity consistency: Throughout which video are entities more consistent (e.g., clothes do not change without a change described in the caption)?
Background consistency: In which video is the background more consistent (e.g., the room does not change without a change described in the caption)?
Action realism: In which video do actions look more realistic (e.g., according to physics)?
An example of our interface is shown in App. D.3. For each evaluation, humans are provided with the full text story; and each text prompt is rendered as a video below each generated video to facilitate mappings of sentences to their corresponding outputs. For the tasks of action execution and story continuation, we also provide the conditioning video but add a red border around it to easily identify its end. The models are anonymized and the output pairs are randomly ordered (left vs. right) for each presentation to a rater. We randomly sample 100 stories, and ask three raters to judge each pair.
3.4 Automatic Metrics
Human evaluation will always be the gold standard of generative models' evaluation. However, it is both time-consuming and expensive to conduct. To help researchers and practitioners quickly deploy and evaluate new systems, different automatic metrics that correlate fairly well with human evaluations have been proposed over the years, such as FID [49] and FVD [50], as well as CLIP [51] for the recent text-guided models. We consider the following automatic metrics for STORYBENCH.
Frechet Image Distance (FID): The FID score evaluates the quality of generated images (i.e., frames here) by comparing how similar their features are to features of real images. We consider both the standard features given by InceptionV3 [52], as well as features from a stronger CLIP (ViT-L/14).
Frechet Video Distance (FVD): FVD extends the idea behind FID to videos: a generative model must capture the underlying distribution of real-world videos. We consider both the standard features from I3D [53], and features from InternVideo-MM-L-14 [54], a state-of-the-art, video-text model.
Video-Text Matching (VTM): We estimate the alignment between generated videos and their text descriptions with the similarity of the visual and textual features given by a multimodal model. We consider both the average cosine similarity of each frame with the corresponding caption given by CLIP (ViT-L/14), and the cosine similarity of a video with its caption given by InternVideo-MM-L-14.
Perceptual Quality Assessment (PQA): We also measure perceptual quality, which aims at predicting the average human subjective perception of a video. For this, we use DOVER [55], a state-of-the-art model trained to predict the mean opinion score of user-generated videos.
Similarity of Images (SIM): For the tasks of action execution and story continuation, a model is conditioned on a ground-truth video, and asked to continue it for the exact same number of frames as the original one. We can hence directly compare each generated frame with its ground-truth match. We compute frame cosine similarity from the normalized features extracted to compute FID.