SlowGuess/NeurIPS_2023_OCR huggingface dataset:
3.1.4 Stage IV: Manual  $\mathcal{Q}\mathcal{A}\mathcal{W}$  Curation
While LLM filtering ensures that the generated  $\mathcal{QA}$  relates to the video content, it's also necessary to ensure the veracity and a long temporal certificate length for every generated  $\mathcal{QAW}$ . This is achieved through a two-step manual curation process.
In the first round of curation, annotators are tasked with three primary responsibilities: (A) First, they verify that  $\mathcal{Q}$  is well-formed and  $\mathcal{A}$  is indeed the correct answer to  $\mathcal{Q}$ . (B) Next, they confirm that all the  $M$  distractors,  $\mathcal{W}$ , are indeed wrong answers to  $\mathcal{Q}$ . (C) Finally, they ensure that the temporal certificate length for answering  $\mathcal{Q}$  is at least 30 seconds.
A  $\mathcal{Q}\mathcal{A}\mathcal{W}$  is discarded if any of these three conditions are not met. This reduces the number of admissible questions by a factor of about  $4\times$  to  $5\times$  within the first round itself. Next is a second round of re-curation, to reinforce the conditions and guarantee data of the highest quality. We find that more than  $97\%$  of the questions that pass the first round also pass the second round, speaking to the efficacy of the curation process. A crucial aspect of ensuring that the question assesses very long-form video-language understanding capabilities is the notion of temporal certificate length (condition (C) above), which we describe next. The detailed procedures for onboarding and training the human annotators, as well as the instructions for the curation process are provided in the supplementary.
3.2 Temporal Certificates
We define the temporal certificate of a given video in a video understanding task to be the minimum set of subclips of the video that are both necessary and sufficient to convince a human verifier that the marked annotation for that data (such as timestamps in temporal activity localization, class label in activity recognition or, the correct option in multiple-choice question-answering) is indeed correct, without having to watch the rest of the clip outside of the certificate set (Fig. 2). Naturally, we define certificate length to be the sum of the temporal lengths of the sub-clips present in the certificate set.
Meta-rules. Datasets often have implicit rules that apply uniformly across the entire dataset. We call these conventions meta-rules and allow the human verifier to be well aware of them. For example, in temporal action localization datasets [29], an implicit assumption is that the action to be localized in a contiguous sub-clip and hence can be uniquely determined by the start and end timestamps. Since this rule is valid for all data, we consider it to be a meta-rule.
A comprehensive understanding of meta-rules of a dataset is necessary for accurate estimation of the certificate set, and hence the certificate length. Otherwise, a spuriously long certificate might be necessary to ensure the veracity of the marked annotations. For example, consider the task of action classification on Kinetics-400. A valid meta-rule to be made available to the human verifier in this case is the mutual exclusivity of action classes i.e., each data point can belong only to one of the 400 classes present in Kinetics-400. Without this understanding, given, say a 10-second clip of a human skiing, the certificate set needs to necessarily encompass the entire 10 seconds since otherwise the human verifier might not be convinced that all of the other 399 actions are not occurring in the clip. However, with the knowledge of the label exclusivity meta-rule, the certificate length will be drastically reduced to just a fraction of a second since just observing the action of skiing in a few frames is sufficient for the human verifier to out-rule all other action classes.
Certificate Conventions. For small certificate lengths, it is difficult for humans to estimate the exact sub-clip timestamps to be included in the certificate set. Hence, we choose to have a minimum length of 0.1 second for a certificate. Further, in the case of two non-contiguous certificates, we collapse them into one if their closest ends are  $< 5$  seconds apart. In cases where a fact needs to be verified at several places throughout the video, we let the annotator make a reasonable judgment for the length of the certificate to be included as long as it follows the above conditions.
4 Benchmarking EgoSchema
4.1 Evaluating Certificate Lengths
Fig. 3 presents certificate lengths for a spectrum of tasks spread across 15 different datasets such as, action classification (Kinetics [7], Something-Something [19], UCF101 [46], HVU-Action [12]), detection (AVA [23]), relationship classification (LVU [51]), concept classification (HVU-Concept [12]), video classification (Youtube-8M [1]), Question-Answering (NextQA [53], AGQA [22], NextQA [53], IVQA [34], MSRVTT [55], ActivityNet-QA [63], EgoSchema). For EgoSchema we benchmark the certificate length for 5 hours of video data  $(100Q\mathcal{A}\mathcal{W})$  chosen randomly. For each other dataset, we ensure that (A) each annotated label class (if applicable) has at least 1 data sample evaluated and, (B) at least two hours of human effort is applied. Fig. 2 shows the histogram of estimated EgoSchema temporal certificate lengths for the 100 clips.
Fig. 3 plots the certificate length against the actual clip length. We observe that EgoSchema has temporal certificate length  $5.7 \times$  longer than the second longest certificate length dataset, and  $10 \times$  to  $100 \times$  longer than all other video understanding datasets.
4.2 Evaluating Multiple-choice Question Answering on EgoSchema
In Table 6, We benchmark several state-of-the-art video-language models, with the intention of adding more models in the future, in a Zero-shot question-answering setting on EgoSchema. We evaluate each model in at least two settings. First is the conventional inference setting, where the model is assessed based on the same number of frames it was trained with. And second is a less challenging setting, where the model is tested on the maximum number of frames possible to execute inference with, using an 80G A100, without exceeding the GPU memory capacity. In both settings, frames are sampled uniformly from the input video clip.
FrozenBiLM [58] adapts frozen multi-modal encoders trained on web-scale data for the task of question answering and achieves state-of-the-art zero-shot QA accuracy across 8 video question-answering datasets. We choose the How2QA FrozenBilM model under both 10 and 90 frames.
VIOLET [14] a masked token modeling-based video language transformer that performs competitively on a variety of video-language tasks. We evaluate four of the best VIOLET models that are finetuned on different tasks for both 5 and 75 frames and choose the model with the best overall accuracy. More details are in supplementary.
mPLUG-Owl [60] proposes a training strategy to add image & video modality to pretrained
Figure 6: Benchmarking Zero-shot QA on EgoSchema
Model	Release	Inference Params	Evaluation Setting	QA Acc
Choosing the correct A uniformly at random	20.0%
FrozenBiLM [58]	Oct 2022	1.2B	10 frames	26.4%
90 frames	26.9%
VIOLET [14]	Sept 2022	198M	5 frames	19.9%
75 frames	19.6%
mPLUG-Owl [60]	May 2023	7.2B	1 frame	27.0%
5 frames	30.2%
10 frames	29.7%
15 frames	28.7%
30 frames	20.0%
InternVideo [49]	Dec 2022	478M	10 frames	31.4%
30 frames	31.8%
90 frames	32%
large language models. We adapt mPLUG to facilitate the multiple choice QA by prompting the model with each of the options individually in the format: 'Given question <question text>, is answer <answer text> correct?' along with the video frames. Then, we choose the option with the highest softmax score of the token 'Yes' in the output text. We observe accuracy to be non-monotonic in frame length, and report results in 1 to 30 frames in Table 6.
InternVideo [49] proposes training video-language models jointly with masked video modeling and contrastive learning objectives. By default, InternVideo does not directly support multiple-choice video QA. We adapt the MSRvTT finetuned InternVideo model, which performs zero-shot
multiple-choice tasks, by incorporating the question with each answer choice in the format: 'Question: <question text?> Is it <answer text>'. Then, we choose the option with the highest output score as the prediction. We report results spanning 10 to 90 input frames in Table 6. We observe that performance is monotonic with the number of frames but the gain saturates around just 30 frames.
Human. We also benchmark human performance on multiple-choice question answering task on EgoSchema in Table 7. First, are time pressure settings where the annotators are asked to choose the correct answer under one ('In <1 min') and three ('In <3 min') minutes. Humans can already achieve an impressive  $67.0\%$  accuracy, in under 1 minute! Interestingly, this only slightly increases  $(+1.0\%)$  when allowed three minutes. We believe that this can inform about performance on EgoSchema in limited model inference capacities. We believe this could inform about the frame rate needed for long-form video understanding in future models. Second, we also benchmark human performance using only 1 fps video ('180 frames'). Surprisingly, we observe that just with 1 fps humans can achieve an impressive  $67.2\%$ .
Third, we evaluate human performance in a restrictive setting where the annotator is forced to first watch the video without reading the text, and then answer the question without rewatching the video ('Video  $\longrightarrow$  Text'). Curiously, this achieves better accuracy than the 'No constraint' setting where the annotators are asked to simply answer without any constraints (76.2% vs. 75.0%). A possible hypothesis is that watching the video without text allows the anno
Figure 7: Human Accuracy on EgoSchema
Evaluation Setting	QA Accuracy
180 frames	67.2%
In <1 min	67.0%
In <3 min	68.0%
No constraint	75.1%
Video → Text	76.2%
tator to focus more closely on the video, thereby benefiting performance than the setting where the attention is somewhat divided between the text and video. We believe this will help us understand the performance trade-offs in the early vs. late fusion of video and text modalities for long-form video-language models. All accuracies are estimated over 5 hours of video.
We present EgoSchema, a novel diagnostic benchmark designed for assessing very long-form video-language understanding capabilities of modern multimodal models. We also introduce the notion of a temporal certificate set, a probe that can be applied to a wide array of video tasks and benchmarks for understanding their intrinsic temporal lengths. We estimate temporal certificates of 15 varied datasets and demonstrate EgoSchema to exhibit temporal certificate length approximately  $5.7 \times$  longer than the next longest dataset and  $25 \times$  to  $100 \times$  longer than all other video understanding datasets. We also benchmark several state-of-the-art models on EgoSchema and find their Zero-shot question-answering accuracy to be less than  $33\%$  while humans achieve  $76\%$ . We believe that EgoSchema will play a key role in the development and evaluation of future very long-form video-language models.
Limitations. EgoSchema RGB clips are sourced from Ego4D [20] and inherit Ego4D egocentric video biases. Further, the text is carefully curated for veracity, there are inevitable text data distribution biases that can occur in LLM-generated outputs due to biases present in web-scale LLM training data. Finally, human curation itself is far from perfect and while we perform two rounds of curation to minimize false positives, the collected EgoSchema is most likely to inevitably contain some small mislabelled or ill-formed question-answer sets. We plan to host a crowd-sourced errata board to minimize human curation error over time with the support of the open-source research community.
[1] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan. Youtube-8m: A large-scale video classification benchmark. arXiv preprint arXiv:1609.08675, 2016.
[2] anthropic. Introducing claude, 2023.
[3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proc. ICCV, 2015.
[56] Li Xu, He Huang, and Jun Liu. Sutd-trafficqa: A question answering benchmark and an efficient network for video reasoning over traffic events. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
[57] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proc. ICCV, 2021.
[58] Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. arXiv preprint arXiv:2206.08155, 2022.
[59] Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the ACM international conference on Multimedia, 2022.
[60] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.14178, 2023.
[61] Yunan Ye, Zhou Zhao, Yimeng Li, Long Chen, Jun Xiao, and Yueting Zhuang. Video question answering via attribute-augmented attention network learning. In Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, 2017.
[62] Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum. Clevrer: Collision events for video representation and reasoning. arXiv preprint arXiv:1910.01442, 2019.
[63] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, 2019.
[64] Heeseung Yun, Youngjae Yu, Wonsuk Yang, Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio-visual question answering on 360deg videos. In Proc. ICCV, 2021.
[65] Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Socialiq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[66] Wentian Zhao, Seokhwan Kim, Ning Xu, and Hailin Jin. Video question answering on screencast tutorials. arXiv preprint arXiv:2008.00544, 2020.
EgoSchema Datasheet
EgoSchema is a diagnostic benchmark for assessing very long-form video-language understanding capabilities of modern multimodal systems. While some prior works have proposed video datasets with long clip lengths, we posit that merely the length of the video clip does not truly capture the temporal difficulty of the video task that is being considered. To remedy this, we introduce temporal certificate sets, a general notion for capturing the intrinsic temporal understanding length associated with a broad range of video understanding tasks & datasets. Please see Section 3.2 in the main paper for more details.
The authors created the dataset within the Malik Group at Berkeley AI Research, UC Berkeley. The authors created it for the public at large without reference to any particular organization or institution.
Each instance in the dataset represents a 3-minute video and text that contains a question and five answer options.
EgoSchema has a total of 5063 instances each containing one video, one question, and five answer options. You can see further statistics on the whole data on our website egoschema.github.io.
The video component of our dataset derives from the broader Ego4D dataset. For our research, we selectively extracted non-overlapping three-minute segments from the Ego4D video data, each segment consisting of a minimum of 30 human-annotated narrations (where each narration refers to a timestamped sentence). Detailed statistic of the number of viable clips for different possible length and narration density choices is discussed in Supplementary Section 6. The selected subset is very diverse in human behavior as can be seen by the activity statistics presented on egoschema.github.io.
Each instance in our dataset comprises raw mp4 video data, captured at a rate of 30 frames per second and with a high resolution. Accompanying this video data, there are six text elements - one question and five corresponding answer options one of which is marked as the correct answer to the question.
Each instance is associated with a label ranging from 1 to 5 that indicates which of the five answer options is correct.
All instances are complete.
Some instances may have the same video but different questions and answers. It will be indicated by a clip unique identifier in the final dataset.
EgoSchema is designed specifically for zero-shot testing. Its primary purpose is to be able to assess the out of the box long-term video-language understanding capabilities of modern multimodal models.
The dataset was very carefully manually curated to mitigate any incidence of errors within the questions and answers. Although different questions may be posed for the same clip, it is ensured that there is no overlap between any two distinct clips. Further related details are also discussed in the limitations section in the main paper.
Entirety of the dataset will be made publicly available at our project website egoschema.github.io. We will also provide a download tool for preprocessing all the videos such as cutting clips, associating the question/answer text etc. Text will be released in a JSON format, hosted on our github repository. EgoSchema will be publicly released under the Ego4D license, which allows public use of the video and text data for both research and commercial purposes.
Some videos do contain people. However, the Ego4D authors employed an array of de-identification procedures primarily centered on ensuring a controlled environment with informed consent from all participants, and, where applicable, in public spaces with faces and other personally identifiable
information suitably obscured. We strictly import all RGB information from Ego4D without any addition of our own.
No, Ego4D has employed an array of deidentification procedures in order to obscure any personally identifiable information such as people's faces.
The video data, which is directly observable, was procured from the publicly accessible Ego4D dataset. In contrast, the text data was generated through the use of Large Language Models (LLMs) including GPT4, BARD, and Claude. These LLMs employed visual narrations from each video within the Ego4D dataset to generate the corresponding text.
The video and narration data were downloaded in accordance with the official Ego4D guidelines for data access: https://ego4d-data.org/docs/start-here. For the generation of the text data within our dataset, we utilized API access for GPT4 via OpenAI, for BARD via Google, and for Claude via Anthropic. This allowed us to generate three distinct questions for each video clip sampled from the Ego4D dataset. Upon the generation of these questions for each sampled video clip, we implemented a series of filtering procedures including Rule-based filtering, Blind filtering, and Manual curation. See Section 3.1.2 in the main paper for a more detailed explanation.
The video component of our dataset derives from the broader Ego4D dataset. For our research, we selectively extracted non-overlapping three-minute segments from the Ego4D video data, each segment consisting of a minimum of 30 human-annotated narrations (where each narration refers to a timestamped sentence). Detailed statistic of the number of viable clips for different possible length and narration density choices is discussed in Supplementary Section 6.
Our research utilized the services of Quantigo, a specialized data labelling company. The teams of Quantigo employees that were based in Bangladesh were compensated at a rate of 5 dollars per hour, at a wage significantly higher than the market hourly rate in Bangladesh. This was done to ensure fair compensation for the complex tasks performed while also contributing to the highest quality of the work delivered. It's important to note that our collaboration with Quantigo followed ethical guidelines, with the fair treatment of all employees involved and the appropriate respect for their expertise and labor. For exact instructions for human curation, see Supplementary Section 7.
Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
The original videos within the Ego4D dataset were collected across various occasions spanning from 2019 to 2021. As for the EgoSchema, the textual information was collected over several sprints during the first half of 2023 based on the Ego4D narrations.
The video and narration data were acquired in accordance with the official Ego4D guidelines for data access: https://ego4d-data.org/docs/start-here/. The Ego4D authors had in turn ensured consent of the people involved.
Ego4d paper followed several procedures to ensure the preservation of privacy and the upholding of ethical standards. Notably, these procedures included obtaining informed consent from those wearing the cameras and adhering to de-identification requirements for personally identifiable information (PII). Given that the video collection was conducted by Ego4D, we are not in a position to provide specific instructions that were given to the camera wearers. The Ego4D privacy statement is available at https://ego4d-data.org/pdfs/Ego4D-Privacy-and-ethics-consortium-statement.pdf
Ego4d paper privacy procedures have included obtaining informed consent from those wearing the cameras. Given that the video collection was conducted by Ego4D, we are not in a position to provide specific instructions that were given to the camera wearers. See Ego4D privacy statement.
Ego4d paper privacy procedures have included allowing camera users to ask questions and withdraw at any time. Additionally, they were free to review and redact their own video. Given that the video collection was conducted by Ego4D, we are not in a position to provide specific instructions
that were given to the camera wearers. You can find the Ego4D privacy statement at https://ego4d-data.org/pdfs/Ego4D-Privacy-and-ethics-consortium-statement.pdf.
While we recognize the importance of this topic, we would, once more, refer to the Ego4D paper for an in-depth discussion. Ego4D acknowledges the potential privacy risks associated with the use of wearable devices in data collection and has taken several steps such as depersonalizing any sensitive information, blurring out faces and bodies, etc. towards maintaining privacy. The same carries over to the video data in EgoSchema as well. We observed 50 randomly sampled three-minute clips from Ego4D, out of which we found 5 to have un-blurred and non-obstructed human faces (10%), collected with the participant's consent. Broadly, very long-form video understanding is a core capability for agents that are to perceive the natural visual world. Hence, developing datasets such as EgoSchema will be critical to unlocking this key AI capability. Additionally, according to Ego4D privacy statement, all videos from Ego4D were reviewed by an approved member of one of the participant's universities or institutes to identify and assess potential privacy concerns.
Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remainder of the questions in this section.