SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In Section 4, we define an automatic pipeline to transform the original VidLN captions for Oops and UVO into multiple sentences, each approximately describing a single action, and to estimate their corresponding timestamps. Here, we compute some statistics to assess the quality of the stories generated automatically through our algorithmic pipeline by comparing them against human references for the Oops Dev set (1,578 stories).
Figure 9: Statistics of our final dev sets.
Figure 10: Statistics of our final test sets.
Figure 11: Robustness statistics of automated pipeline for story-like data transformation. Left: Number of captions per story. Right: Duration of video segments in seconds.
PROMPT: A man wearing white shorts is jumping on a trampoline.
PROMPT: The man performing a flip.
PROMPT: The man falls when the trampoline falls on the ground.
Figure 12: Example story (subsampled frames) from Oops-CSV.
PROMPT: A baby wearing blue clothes first touches the girl's ice cream while the first girl is eating her ice cream.
PROMPT: The baby turns back.
PROMPT: The baby starts climbing on the back side of the seat.
Figure 13: Example story (subsampled frames) from UVO-CSV.
As shown in Figure 11 (left), the distributions of the number of sentences per story of the two approaches are very similar. In particular, we notice that our method tends to split captions into two or three segments more often than the human annotators, who, more often, prefer not to split them.
The words corresponding to each video segment are very similar between human and automatic stories. To assess this, we consider the subset of captions that have been split into the same number of sentences, so we can compute one-to-one mappings between the human and the algorithmic captions. Here, we observe a  $\mathrm{BLEU}_4$  score of  $63.6\%$  (BLEU<sub>4</sub> measures the overlap of 4-grams in the two captions), indicating a relatively high similarity of generated sentences to human references. We also note that humans were asked to enrich the original Oops and UVO captions with context information (e.g., what other relevant actors are doing while a specific actor is being narrated), which our algorithmic pipeline does not explicitly tackle, leaving room for improvement in future work.
Finally, Figure 11 (right) shows that the resulting duration of the algorithmically generated video segments are slightly longer than human-annotated timestamps.
PROMPT: A white-brown dog is sitting and starts moving towards the person.
PROMPT: A person whose only hand and leg is visible is holding some food in his hand.
PROMPT: The white-brown dog is take food from the person hand and eats, while the camera focus on the dog face.
PROMPT: The person starts rubbing the dog head with his hand.
Figure 14: Example story (subsampled frames) from DiDeMo-CSV.
In this section, we report our full set of results from our baselines on STORYBENCH, in terms of both human evaluations and through automatic metrics. Recall that we append -ZS for results obtained in the zero-shot setting, -ST for single-task fine-tuning, and -MT for multi-task fine-tuning. Each model was fine-tuned for 500K steps in less than a day on  $4\mathrm{x}4\mathrm{x}4$  TPUv4 chips. For every story, each model generates 4 output videos at 8fps using a  $160\times 96$  pixel resolution. We randomly sample one of them for human evaluation (e.g., Figure 16), but report mean and standard deviation for automatic metrics.
E.1 Human Evaluation
Figure 17 shows the results of human evaluation, where each bar displays the number of wins of two given models evaluated side-by-side, as well as the number of ties (in white). For each story, we ask three human raters to compare two models and report the majority vote in Figure 17.
Figure 15: Example of our human rating interface.
PROMPT: The swimmers dive into the water and starts swimming from one end to the another.
Figure 16: Example of generated actions by PHENAKI-GEN-ST and PHENAKI-CONT-ST on DiDeMoCSV. PHENAKI-GEN-ST quickly changes the background, while PHENAKI-CONT-ST correctly synthesizes a person swimming left-to-right without distorting the background. Video subsampled by a factor 4 to be shown here.
Looking at task of action execution on Oops-CSV, we see that our PHENAKI-CONT-ST achieves competitive performance with our PHENAKI-GEN-ZS baseline, with better text adherence, background consistency and action realism. This result is not surprising as most of the actions in Oops are short (less than 5s). It is interesting, however, to see that our annotators find PHENAKI-CONT-ST largely better than PHENAKI-GEN-ST across all criteria. On the other hand, none of these models clearly outperforms others for the most challenging task of story generation.
For the task of story continuation, PHENAKI-CONT-ST typically outperforms both PHENAKI-GEN-ZS and PHENAKI-GEN-ST, especially on Oops-CSV. On UVO-CSV and DiDeMo-CSV, PHENAKI-CONT-ST consistently outperforms PHENAKI-GEN-ST except for entity and background consistency, where human raters often have no preference between the two. Comparing multi-task models, we find that PHENAKI-CONT-MT is always preferred to PHENAKI-GEN-MT; yet PHENAKI-GEN-ZS is a strong baseline, achieving better visual quality than the fine-tuned models.
Figure 17: Results from human evaluation across datasets and tasks.
E.2 Automatic Evaluation
For completeness, Tables 8 to 10 report the performance of our baselines on all tasks and datasets when instead using CLIP to compute FID and SIM, and InternVideo to compute FVD and VTM. We find similar patterns as with other metrics (c.f. Section 6), but also notice that InternVideo (used for FVD and VTM) favors the videos generated by the zero-shot PHENAKI-GEN model.
Action Execution Model (@8 fps)	Oops-CSV	UVO-CSV	DiDeMo-CSV
FIDC↓	FVDIV↓	SIMC↑	PQA↑	VTMIV↑	FIDC↓	FVDIV↓	SIMC↑	PQA↑	VTMIV↑	FIDC↓	FVDIV↓	SIMC↑	PQA↑	VTMIV↑
Zero-shot
PHENAKI-GEN-ZS	94.7±0.52	126.7±0.46	64.9±0.08	5.8±0.03	22.6±0.07	79.2±0.38	85.3±0.41	66.7±0.03	8.5±0.10	23.0±0.03	97.2±0.34	78.0±0.25	64.3±0.08	6.7±0.02	22.9±0.05
Single-Task
PHENAKI-GEN-ST	97.1±0.20	179.4±0.28	64.8±0.04	4.0±0.02	20.0±0.02	97.3±0.18	147.6±0.20	62.5±0.04	4.8±0.05	18.4±0.02	89.3±0.22	120.2±0.59	64.9±0.04	4.5±0.02	20.4±0.03
PHENAKI-CONT-ST	84.5±0.02	171.6±0.58	67.9±0.04	4.8±0.02	19.9±0.01	92.4±0.27	143.2±0.31	64.0±0.09	5.6±0.02	18.7±0.03	82.5±0.22	107.3±0.46	66.8±0.01	5.6±0.01	20.1±0.01
Multi-Task
PHENAKI-GEN-MT	102.8±0.64	179.3±0.63	63.7±0.04	3.8±0.04	20.1±0.04	92.1±0.68	138.9±0.42	63.3±0.03	5.1±0.05	19.2±0.02	88.2±0.44	119.6±0.47	64.3±0.04	4.7±0.06	20.1±0.02
PHENAKI-CONT-MT	86.0±0.52	171.3±0.56	67.4±0.11	4.7±0.01	20.1±0.03	77.9±0.08	126.8±0.24	67.1±0.06	6.8±0.01	19.9±0.02	85.4±0.14	106.5±0.29	66.4±0.07	5.8±0.03	19.9±0.03
Table 8: Results from automatic evaluation metrics on action execution tasks. Best results are in bold. FID and SIM use CLIP, FVD and VTM use InternVideo, and PQA uses DOVER.
Story Continuation Model (@8 fps)	Oops-CSV	UVO-CSV	DiDeMo-CSV
FIDC↓	FVDIV↓	SIMC↑	PQA↑	VTMIV↑	FIDC↓	FVDIV↓	SIMC↑	PQA↑	VTMIV↑	FIDC↓	FVDIV↓	SIMC↑	PQA↑	VTMIV↑
Zero-shot
PHENAKI-GEN-ZS	103.2±0.87	116.6±0.54	63.1±0.05	7.2±0.06	22.5±0.05	82.2±0.51	83.6±0.44	65.9±0.04	9.4±0.09	22.9±0.03	108.2±0.43	87.9±0.41	61.7±0.04	7.3±0.07	22.5±0.10
Single-Task
PHENAKI-GEN-ST	99.6±0.07	181.5±0.74	64.0±0.05	4.3±0.02	19.7±0.02	97.8±0.26	151.1±0.21	62.5±0.03	5.0±0.03	18.2±0.01	90.9±0.18	127.5±0.48	64.2±0.03	4.0±0.02	19.9±0.02
PHENAKI-CONT-ST	89.2±0.30	169.9±0.67	66.3±0.05	5.3±0.04	19.5±0.02	94.1±0.29	147.3±0.75	63.5±0.07	5.7±0.03	18.3±0.02	89.4±0.29	118.1±0.60	64.5±0.05	5.4±0.03	19.4±0.06
Multi-Task
PHENAKI-GEN-MT	105.2±0.26	182.0±0.17	63.0±0.04	4.2±0.02	19.8±0.02	92.8±0.58	141.9±0.22	63.1±0.03	5.2±0.03	18.9±0.02	90.7±0.24	126.4±0.82	63.4±0.02	4.3±0.04	19.7±0.04
PHENAKI-CONT-MT	92.1±0.44	171.4±0.67	65.7±0.09	5.1±0.02	19.8±0.02	80.4±0.17	129.0±0.65	66.3±0.09	7.0±0.02	19.6±0.05	95.5±0.33	120.2±0.23	63.4±0.07	5.5±0.09	19.0±0.01
Table 9: Results from automatic evaluation metrics on story continuation tasks. Best results are in bold. FID and SIM use CLIP, FVD and VTM use InternVideo, and PQA uses DOVER.
Story Generation
Model (@8 fps)	Oops-CSV	UVO-CSV	DiDeMo-CSV
FIDC↓	FVDIV↓	SIMC↑	PQA↑	VTMIV↑	FIDC↓	FVDIV↓	SIMC↑	PQA↑	VTMIV↑	FIDC↓	FVDIV↓	SIMC↑	PQA↑	VTMIV↑
Zero-shot
PHENAKI-GEN-ZS	117.2±0.90	113.0±0.54	N/A	8.1±0.03	22.9±0.11	97.5±0.89	88.2±0.68	N/A	10.0±0.06	22.6±0.14	115.6±0.38	91.1±0.46	N/A	7.6±0.08	23.0±0.06
Single-Task
PHENAKI-GEN-ST	103.4±0.28	181.4±0.19	N/A	4.2±0.02	19.6±0.03	99.2±0.22	151.7±0.45	N/A	4.9±0.01	18.0±0.01	92.3±0.07	128.6±0.44	N/A	4.0±0.01	20.1±0.04
PHENAKI-CONT-ST	109.1±0.89	167.9±0.95	N/A	5.4±0.03	17.9±0.03	100.6±0.47	149.6±0.24	N/A	5.4±0.02	17.4±0.02	96.1±0.11	124.6±0.86	N/A	5.4±0.05	18.9±0.02
Multi-Task
PHENAKI-GEN-MT	107.4±0.71	180.0±0.52	N/A	4.1±0.02	19.8±0.06	94.8±0.25	143.0±0.28	N/A	5.1±0.03	18.8±0.01	92.0±0.20	127.3±0.46	N/A	4.3±0.06	19.8±0.01
PHENAKI-CONT-MT	114.3±0.12	171.0±0.53	N/A	5.0±0.11	18.0±0.06	99.8±0.28	132.7±0.52	N/A	6.3±0.08	17.5±0.05	105.6±0.26	125.9±1.03	N/A	5.2±0.03	18.3±0.06
Table 10: Results from automatic evaluation metrics on story generation tasks. Best results are in bold. FID uses CLIP, FVD and VTM use InternVideo, and PQA uses DOVER.