SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Karttikeya Mangalam
Raiymbek Akshkulakov
Jitendra Malik
{mangalam, raiymbek, malik}@eecs.berkeley.edu
egoschema.github.io
We introduce EgoSchema, a very long-form video question-answering dataset, and benchmark to evaluate long video understanding capabilities of modern vision and language systems. Derived from Ego4D, EgoSchema consists of over 5000 human curated multiple choice question answer pairs, spanning over 250 hours of real video data, covering a very broad range of natural human activity and behavior. For each question, EgoSchema requires the correct answer to be selected between five given options based on a three-minute-long video clip. While some prior works have proposed video datasets with long clip lengths, we posit that merely the length of the video clip does not truly capture the temporal difficulty of the video task that is being considered. To remedy this, we introduce temporal certificate sets, a general notion for capturing the intrinsic temporal understanding length associated with a broad range of video understanding tasks & datasets. Based on this metric, we find EgoSchema to have intrinsic temporal lengths over  $5.7 \times$  longer than the second closest dataset and  $10 \times$  to  $100 \times$  longer than any other video understanding dataset. Further, our evaluation of several current state-of-the-art video and language models shows them to be severely lacking in long-term video understanding capabilities. Even models with several billions of parameters achieve QA accuracy less than  $33\%$  (random is  $20\%$ ) on the EgoSchema multi-choice question answering task, while humans achieve about  $76\%$  accuracy. We posit that EgoSchema, with its long intrinsic temporal structures and diverse complexity, would serve as a valuable evaluation probe for developing effective long-term video understanding systems in the future. Data and Zero-shot model evaluation code are open-sourced under the Ego4D license at egoschema.github.io.
We introduce EgoSchema, a diagnostic benchmark for assessing very long-form video-language understanding capabilities of modern multimodal systems. Understanding long natural videos requires a host of interconnected abilities such as action and scene understanding, perceiving and tracking object states, long-term visual memory, abstract reasoning, hierarchical information aggregation, and more. Shown in Fig. 1 is an exemplar of the curated EgoSchema dataset. Consider the visual cognitive faculties involved in answering the question: 'What is the overarching behavior of C and the man in the video?' First, is the spatial recognition capabilities for disambiguating the
What is the overarching behavior of C and the man in the video?
C teaches the man game rules but the man seems distracted and is not paying attention
The man teaches C how to play the card game while organizing the deck for future games
C and the man are playing a card game while keeping track of it in a notebook
C shows the man how to properly shuffle cards while the man plays them
The man shows C a new card game while C takes notes for future reference
Full Video Link: youtu.be/Tp4q5GeHVMY
Full Video Link: youtu.be/DllyVccQPbg
Observe the video in terms of characters' actions and interactions. How do these shifts contribute to the overall narrative?
The video displays a profound sense of conflict and tension arising between the characters
The man is showing C the issues that need fixing in the apartment in a professional manner
Both the characters display an increasingly urgent need to solve an issue in the apartment.
C and the man admire and interact with several objects in the apartment that look beautiful.
Figure 1: The EgoSchema dataset contains over 5000 very long-form video language understanding questions spanning over 250 hours of real, diverse, and high-quality egocentric video data. Each question requires choosing the correct answer out of five choices based on a three minute long video clip. The questions are manually curated to require very long temporal certificates (§3.2). EgoSchema median certificate length is about 100 seconds, which is  $5 \times$  longer than the closest second dataset and  $10 \times$  to  $100 \times$  longer (Fig. 3) than any other video understanding dataset. State-of-the-Art video-language models consisting of billion of parameters achieve very low accuracy ( $< 33\%$ ) in Zero-shot evaluation (random is  $20\%$ ) while humans achieve about  $76\%$ . ‘C’ refers to the camera wearer. Visualized clips are available at egoschema.github.io/explorer.
Actions and interactions are casual and relaxed, reflecting a comfortable environment.
referred character 'C' (camera wearer) and 'the man' as well as the present objects such as 'cards', 'notebook', deck as so on. Next is short-term temporal recognition capabilities of understanding the atomic actions and movement of the characters such as 'playing', 'taking notes', 'shuffling' etc. Built upon these are the capabilities for visually understanding the mental states such 'distracted', 'attention' and social dynamics such as 'teaching', 'showing'. Next are medium-term actions such as 'organizing the deck' or 'keeping track'. Finally, long-term reasoning capabilities need to be employed for abstracting the 'overarching behavior' of the video from all the low-level signals to be able to rule out all the other wrong options and conclude option 3 to be correct. Note that even for humans, it is impossible to answer the illustrated questions with only the shown 9 uniformly sampled frames from the three-minute video (Fig. 1).
While there have been some prior attempts to formulate long-form video tasks [51, 44], they broadly tend to fall into two failure modes. The first failure mode stems from the difficulty of capturing the explosive diversity of human behavior in narrow pre-defined label spaces that leading unduly narrow and oddly specific tasks, such as like ratio or relationship prediction [51]. Hence, we propose to probe video systems capturing the rich complexity of long-form video with something just as rich and complex – natural language. However, natural language outputs are notoriously difficult to evaluate with popular metrics such as BLEU [40] and ROUGE [33] having well-known shortcomings [6]. Hence, we propose to evaluate language understanding as a multiple-choice question-answering task, thereby using the well-defined benchmark metric of overall question-answering accuracy.
The second failure mode for a long-term video task is that the proposed task happens to actually be a short-term one - only disguised as a long-term task. To measure the intrinsic "long-term" nature
Figure 2: We introduce the notion of a temporal certificate set (top, §3.2), a tool to measure the intrinsic temporal length of a benchmark and show the EgoSchema certificate length distribution (bottom, §4.1) for randomly chosen 100 clips.
Figure 3: Certificate Length across video datasets for a broad spectrum of tasks such as action classification, detection, relationship classification, concept classification, video classification, and multiple choice question-answering. §4.1 details the precise operationalizations.
of a video understanding task, we propose the notion of temporal certificate length [4]. Intuitively, certificate length (§3.2) is the length of the video a human verifier needs to observe to be convinced of the veracity of the marked annotation. The idea of temporal certificates is not limited only to question-answering or vision-language tasks but is applicable to several video understanding tasks, including pure vision tasks such as action classification, detection, or even temporal action localization.
Based on the length of the temporal certificate, we propose the following temporal understanding taxonomy for video tasks: Datasets with certificate length in the order of 1 second are termed short video tasks. Next, we name datasets with certificate length in the order of 10 seconds as, long-form video tasks. Finally, datasets with certificate length in the order of 100 seconds are termed as, very long-form video tasks. Fig. 3 presents estimates of the certificate lengths for a variety of datasets plotted against the temporal length of the video clip. We observe that the temporal certificate length is quite weakly correlated with the length of the video clip. This is due to the intentional design choice in defining the certificate set, which decouples the task of searching or retrieving the relevant sub-clip from a bigger clip from the task of visually understanding the retrieved sub-clip. And in this manner, using temporal certificate length as a metric for measuring the intrinsic temporal hardness of a dataset, avoids the failure mode of formulating an implicitly short-term task disguised as a long-term one. Section 3.2 details precise operationalizations for estimating the temporal certificate sets.
In summary, our contributions are three-fold. First, we propose the notion of temporal certificates, a broadly applicable notion that measures the intrinsic temporal hardness of clips in a video understanding dataset. We estimate temporal certificate lengths for a broad variety of existing datasets and show that EgoSchema has a median temporal certificate of about 100 seconds, which is  $5 \times$  longer than the dataset with the second longest certificate length [51], and  $25 \times$  to  $100 \times$  longer than all other existing video understanding datasets (with or without language). Second, building upon the notion of temporal certificates, we introduce EgoSchema, a diagnostic benchmark for assessing the very long-form video understanding capability of multimodal video-language systems. Third, we benchmark both state-of-the-art video-language systems and humans in Zero-shot settings on EgoSchema to find that even the most advanced current video-language understanding systems consisting of billion of parameters achieve very low accuracy in long-from multiple-choice question-answering ( $< 33\%$ ) while humans achieve about  $76\%$  accuracy in the unconstrained setting.
Figure 4: EgoSchema data pipeline. Stage I filters the suitable Ego4D RGB videos and narrations for question-answer generation (§3.1.1). Stage II uses narrations in a chained LLM prompting (§3.1.2) procedure to generate multiple QAW triplets per three-minute video clip (§3.1.2). Stage III performs pre-filtering with rule-based and LLM-based logic (§3.1.3). Finally, Stage IV involves two rounds of human curation on filtered QAW for selecting very long-form video-language understanding data (§3.1.4). The stage width ratios are indicative of the filter selection ratios.
Video Question-Answering Datasets. Visual Question-Answering [3] is a popular video-language task with several large internet-scale datasets for video-language pre-training such as Ego4D [20], HowTo100M [35] and HowToVQA69M [34]. However, as the scope and size of pre-training datasets and models soar, it becomes critical to construct evaluations for assessing the model capabilities on various axes. Hence, many smaller datasets have been proposed for evaluating different aspects of video-language understanding such as compositional reasoning [21, 22], causal and common scene comprehension [53], instruction understanding [34, 57], video description ability [55], dynamic environments understanding [15], complex web video understanding [63], situated reasoning [50], spatiotemporal reasoning [28], social intelligence [65], dynamic neuro-symbolic reasoning [62], external knowledge-based reasoning [16] and many more [37, 61, 42, 10, 9, 13, 45, 56, 30, 31, 8, 64, 11, 32, 59, 66, 52, 24]. How2VQA69M [34] and iVQA [34] have leveraged HowTo100M [35] ASR text for generating questions. However, unlike Ego4D narrations that are used in EgoSchema, ASR text does not necessarily describe the visual elements in the scene. Hence, questions can suffer from biases where a key required information is visually absent. [25]. Additionally, generated question-answers also have quite short certificate lengths (iVQA in Fig. 2) due to the local nature of the ASR text.
Long-form Video Understanding Datasets have been very sparsely explored in prior works. [51] posits a long-form video understanding benchmark but the proposed tasks are unduly narrow and specific, such as the 'like' ratio and view count prediction. Also, [51] average certificate length is about  $5.7 \times$  smaller than EgoSchema.
[36] proposes a dataset for benchmarking efficient video inference consisting of frame-wise object mask annotations from Mask-RCNN [26] but without any long-term annotations. [43] introduces a dataset of about 111 hours of video sourced from Kinetics-400 [7] for generic event boundary detection. While the task itself requires comprehensive understanding, the video clip length is only 10 seconds long, with temporal certificates (§3.2) being much shorter. [47] proposes a question-answering dataset based on long movie clips but due to the open-ended nature of questions, successful approaches tend to neglect the visual data and are biased purely with approaches using additional text such as story lines. [44] proposes MAD, a language grounding dataset with an average clip of 110 minutes. However, the length of the retrieved clip is quite short (average 4.1 seconds) thereby resulting in a temporal certificate (§3.2) only a few seconds long. Further, MAD [44] and several other movie-based datasets [27, 48, 54] do not release any video data because of copyright issues. In contrast, EgoSchema has an average certificate length of about 100 seconds. Further, EgoSchema
Question Generation Prompt
I want you to act as a teacher in the class called "Long-term video understanding". I will provide video action narrations and their timestamps and you will generate three highly difficult and diverse questions .... (continues)
...Please label each question as "Question 1,2,3:[full question]"
Timestamps and narrations:  $<$  data >
Option Generation Prompt
I want you to create a difficult multiple-choice exam that tests above student abilities based on the three questions I just provided. Each question should have five similar open-ended but short answers .... (continues)
... Print each correct answer exactly as "Correct answer: [full answer]"
Timestamps and narrations: <data>
Blind Filtering Prompt
Given a question and five answer choices please try to provide the most probable answer in your opinion. Your output should be just one of A,B,C,D,E and nothing else.
Question:  $<$  question  $>$
Option A: <distractor answer>
Option B: <distractor answer>
Option C: <distractor answer>
Option D: <correct answer>
Option E: <distractor answer>
Figure 5: An abridged example of the generation and filtering prompts used in the EgoSchema data generation pipeline (§3). Full versions are provided in the supplementary.
will be publicly released under the Ego4D license, which allows direct public use of the video and text data for both research and commercial purposes.
3 Collecting EgoSchema
Collecting video and language datasets, even without a focus on very long-form video is quite challenging. Manually collecting, observing, and annotating videos with free-form language, in contrast to using images and pre-defined label categories, is both labor-intensive and time-consuming and thereby quite expensive. In addition to burgeoning cost, ensuring visual data diversity and minimizing visual and linguistic bias while ensuring high quality of marked annotations also contribute to the overall difficulty. All these factors get severely more challenging for long-form videos.
In this work, we propose a staged data collection pipeline (Fig. 4) utilizing existing large-scale but short-term video datasets, rule-based filtering procedures, and exciting new capabilities afforded by LLMs to significantly lighten the burden on human annotators. We use the proposed pipeline for curating EgoSchema, a high-quality and diverse very long-form video question-answering dataset. Associated datasheets [17] and data cards [41] for EgoSchema are provided in the supplementary.
3.1 EgoSchema Pipeline
3.1.1 Stage I: Raw Data Filtering
Ego4D [20] has over 3670 hours of RGB video spread consisting of over 3.85 million narration instances covering over 1,772 unique verbs (activities) and 4,336 unique nouns (objects) [20]. The narrators are instructed to continuously pause and describe everything that the camera wearer ('C') does. This creates dense and precise narrations that accurately describe the visuals.
Naturally, the collected video has non-uniform length and narration density. Since we would like to standardize the clip length for evaluation and have sufficiently rich narrations to allow interesting question-answer pairs to form in later stages, we filter the data based on the length and narration density. We choose to filter for non-overlapping three-minute clips each with at least 30 human annotated narrations (each narration is a timestamped sentence) to build EgoSchema. Detailed statistic of the number of viable clips for different possible length and narration density choices is discussed in supplementary.
3.1.2 Stage II: Question Answer Generation
The filtered narrations are processed with a capable LLM to generate  $N$  Question-Answer triplets  $(\mathcal{Q},\mathcal{A},\mathcal{W})$ , each consisting of the question  $\mathcal{Q}$ , the correct answer  $\mathcal{A}$ , and  $M$  wrong answers  $\mathcal{W}$ , per clip. To achieve this, we experimented with several LLM inference call chaining procedures with trade-offs between quality and cost of generation that are briefly described next.
One-shot is the simplest prompting procedure to prompt for all  $N$  instances of  $\mathcal{Q}\mathcal{A}\mathcal{W}$  in one inference call. This is the most cost-efficient option but we found the generations to be of significantly low quality. The generated  $\mathcal{Q}$  often are very similar to each other and the generated  $\mathcal{A}\mathcal{W}$  have a very high false positive rate for the correct answers as well as a false negative rate for the wrong answers.
N-shot is the next natural prompting procedure where we generate one  $\mathcal{Q}\mathcal{A}\mathcal{W}$  per LLM inference call. This significantly improves the false positive and false negative rates but since the generated  $\mathcal{Q}$  are independent and generated with the same prompt, they still tend to be very similar (comparable to one-shot), even at higher sampling temperatures. Further, the cost of generation also scales with  $N$ .
QAW-shot generates each of the  $N$  questions  $\mathcal{Q}$  in one inference call, followed by another inference call for generating  $N$  correct answer  $\mathcal{A}|\mathcal{Q}$  and finally,  $N\times M$  wrong answers,  $\mathcal{W}|\mathcal{Q},\mathcal{A}$ . Since each of the  $N\mathcal{Q}$  is generated jointly, they can be forced to be distinct with appropriate prompting. Similarly, the generated  $\mathcal{A}$  and  $\mathcal{W}$  can also be made distinct. However, this requires 3 chained LLM inference calls, and generation failures in earlier calls cascade steeply.
$\mathbf{Q}(\mathbf{AW})$ -shot generates each of the  $N$  questions  $\mathcal{Q}$  in one inference call, followed by a final inference call for generating all the  $N$  correct and  $N\times M$  incorrect answers in one go  $\mathcal{A}$ ,  $\mathcal{W}|\mathcal{Q}$ . It enjoys the same uniqueness properties as QAW-shot while having just two chained calls, making it both  $30\%$  cheaper and less prone to generation failure cascading. Further, between Q(AW)-shot and QAW-shot, we observe Q(AW)-shot to have a higher generated  $\mathcal{A}$  quality, perhaps since LLM can jointly model  $\mathcal{W}$  while generating  $\mathcal{A}$ . We choose this to be our main method of choice for generating  $\mathcal{Q}\mathcal{A}\mathcal{W}$ .
Prompt for imputing narrations into the LLM has a tremendous effect on the quality of generated  $\mathcal{Q}\mathcal{A}\mathcal{W}$ . We experiment with several seed prompts for each of which we inspect the quality of the  $N$  generated  $\mathcal{Q}\mathcal{A}\mathcal{W}$  for 10 clips. Based on this we iteratively improve the seed prompts manually in a zeroth order optimization fashion. In total, we experiment with a total of about 85 prompts in this fashion to arrive at our final EgoSchema prompts -  $\mathcal{P}_{\mathcal{Q}}$  for generating  $N\times \mathcal{Q}$  questions and  $\mathcal{P}_{\mathcal{AW}}$  for generating all remaining options  $(\mathcal{A}\mathcal{W})|\mathcal{Q}$ . While we fix the  $\mathcal{P}_{\mathcal{Q}}$  prompt, we use multiple  $\mathcal{P}_{\mathcal{AW}}$  prompts so as to avoid any unintended bias in the options. Fig. 5 shows an abridged example of  $\mathcal{P}_{\mathcal{Q}}$  and  $\mathcal{P}_{\mathcal{AW}}$ , full versions available in supplementary material.
Choice of LLM is extremely crucial for obtaining interesting long-form  $\mathcal{Q}$  and generating hard negatives for  $\mathcal{W}$ . With weaker LLMs, the  $\mathcal{Q}$  diversity across video clips remains narrow, and  $\mathcal{W}$  tends to be either obviously wrong or, too similar to  $\mathcal{A}$  and thus a false negative. While we experimented with both GPT-3 [5] and ChatGPT [38] but only found good quality generated  $\mathcal{Q}\mathcal{A}\mathcal{W}$  at a high enough rate with GPT-4 [39], Bard [18], and Claude [2]. For details please see supplementary.
We generate  $N = 3$  questions per three-minute clip as well as  $M = 4$  wrong answers to every question in addition to the correct answer. We observe that larger  $N$  or  $M$  tends to generate similar questions and wrong answers putting unnecessary pressure on Stages III and IV for filtering.
3.1.3 Stage III: Generated Question Answer Filtering
While Stage II produces several high-quality  $\mathcal{Q}\mathcal{A}\mathcal{W}$ , even the best LLM generations are prone to output format aberrations, hallucinations, and sometimes plain false outputs. Further, despite specific pinpointed prompts (Fig. 5), LLMs can fail to comply. Since, we want to ensure EgoSchema to be extremely high-quality and accurate, we set up several filtering rounds to ensure the correctness and high difficulty of questions.
Rule-based filtering. Keywords from the prompts such as 'long-term', 'narrations', 'timestamp' etc. can sometimes bleed into the generated  $\mathcal{QAW}$  which are then discarded. The output generations can also fail to parse according to a specified format and are also then discarded and the concerned  $\mathcal{QAW}$  is regenerated.
LLM-based filtering. While rule-based filtering weeds out logic errors, we would like to further enrich  $\mathcal{Q}\mathcal{A}\mathcal{W}$  before employing human labor. For example, we aim to ensure EgoSchema requires grounded visual reasoning to solve, and hence questions should not be answerable ungrounded, without carefully observing the video. Hence, we develop a "blind" baseline.
Blind filtering baseline employs LLM to guess the correct answer based on the question, without having access to the video narrations conditioned on the shown filtering prompt (Fig. 5). All such ungrounded questions that can be answered blindly are filtered out. This also ensures that generated  $\mathcal{W}$  are indeed relevant and plausible answers to  $\mathcal{Q}$ , since otherwise, the LLM would be able to guess  $\mathcal{A}$  based only on the setting of  $\mathcal{Q}$ . Note that this is overly restrictive since it is possible that a question is guessed correctly through chance and is not necessarily ungrounded. However, we choose to optimize precision over recall since the amount of filtered  $\mathcal{Q}\mathcal{A}\mathcal{W}$  is still large enough.
No- $\mathcal{Q}$  baseline. We also experimented with a No- $\mathcal{Q}$  baseline, where the LLM is prompted to guess the correct answer using the narrations but without the question  $\mathcal{Q}$ . This ensures that the wrong answers are relevant and plausible to the video clip. However, we found this baseline to have near random accuracy ( $\sim 20\%$ ), highlighting the efficacy of Stage II. Hence, we decided to not use this filter in the final pipeline. Additional details including the full prompt are in supplementary.