SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The set of generated questions and answers from output was filtered by those LLMs and finally curated by humans. A detailed description can be found in Section 3. There was no preprocessing done on the video clips sampled from Ego4D.
Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
Human curation was employed to rectify errors in the question-answer sets, particularly cases where the identified correct answer was wrong or a wrong answer was actually correct. Given the crucial role of this step in ensuring the accuracy of our dataset, we do not find it necessary to release a version of the dataset prior to human curation. However, all the discarded "raw" data is indeed also saved.
Is the software used to preprocess/clean/label the instances available? If so, please provide a link or other access point.
The APIs for the Large Language Models (LLMs) are publicly accessible. The prompts for filtering and instructions for human curation are provided in Supplementary Section 5 and Supplementary Section 7 respectively. Additionally all necessary code for generation, filtering etc. is provided in the supplementary materials.
The dataset will be made publicly available and can be used for both research and commercial purposes under the Ego4D license.
How will the dataset be distributed (e.g., tarball on website, API, GitHub) Does the dataset have a digital object identifier (DOI)?
The dataset will be distributed as a JSON file describing the unique identifier for each clip, the associated question, the five answer options, the label, and additional clip information that facilitates the tracing of the clip back to the original Ego4D data, such as the Ego4D video identification of the clip's source video, among other details. In addition, download tools to acquire and pre-process the video RGB data will also be provided on our website.
The full dataset will be made available upon the acceptance of the paper before the camera-ready deadline.
Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
EgoSchema will be publicly released under the Ego4D license, which allows direct public use of the video and text data for both research and commercial purposes.
The authors of the paper will be maintaining the dataset, pointers to which will be hosted on github repo https://github.com/egoschema/EgoSchema along with the code for download and preprocessing tool, with the actual data hosted either on Amazon AWS as an S3 bucket or as a google drive folder.
We will post the contact information on our website. We will be available through github issues as well as through email.
We will host an erratum on the Github repo in the future, to host any approved errata suggested by the authors or the video research community.
Yes, we plan to host an erratum publicly. There are no specific plans for a v2 version, but there does seem plenty opportunities for exciting future dataset work based on EgoSchema.
N/A There are no older versions at the current moment. All updates regarding the current version will be communicated via our website.
If others want to extend/augment/build on/contribute to the dataset, is there a mechanism for them to do so? If so, please provide a description. Will these contributions be validated/verified? If so, please describe how. If not, why not? Is there a process for communicating/distributing these contributions to other users? If so, please provide a description.
Contributions will be made possible using standard open-source tools, submitted as pull requests to the relevant GitHub repository. Moreover, we will provide information on how to trace sampled clips back to their original source within the Ego4D dataset. This will enable users to access additional Ego4D data, such as narrations, summaries, and object detections, as applicable.
Full Prompts
Here are some of the prompts we developed for generating EgoSchema.
5.1 Set A
5.1.1 Question prompt
I want you to act as a teacher in the class called "Long-term video understanding". I will provide video action narrations and their timestamps and you will generate three highly difficult and diverse questions for your students about the high-level details in the video. You want to test students' following abilities:
Ability 1: Students' ability to summarize and compare long parts of the video
Ability 2: Students' ability to compress information from the video rather than just listing the actions that happened in the video.
Ability 3: Students' ability to identify the most important parts of the video.
Your questions should not mention any particular timestamps or narrations. Remember to make sure the correct answers to your questions do not list information from the narrations but compress them in a concise conclusion.
Examples of good and difficult questions:
"What is the main action of the video?"
"Why did C do action ...?"
AVOID the following types of questions:
"When . . .?"
"How many . . .?"
"How much . . .?"
When announcing the question please label each question as "Question 1,2,3: [full question]"
Timestamps and narrations:
O - C stares at the lamp
1 - C looks around the apartment
3 - C talks to a man
6 - C walks around the apartment
10 - C talks to a man
11 - C looks around the apartment
14 - a man plays the guitar
20 - a man walks around the apartment
21 - a man plays the guitar
21 - C walks around the apartment
22 - C stares at the window
24 - C looks around the apartment
27 - C talks to a man
29 - C picks chips from the table
32 - C looks around the apartment
36 - C stares at a man
36 - a man plays the guitar
37 - C walks around the apartment
41 - C sits on the sofa
43 - C stares at a man
44 - a man plays the guitar
49 - a man climbs up the sofa
50 - a man plays the guitar
62 - a man climbs down the sofa
63 - a man walks around the apartment
64 - a man plays the guitar
93 - C stares at the table
96 - a man plays a guitar
111 - C stares at the table
112 - a man walks around the apartment
114 - C stares at a man
115 - a man lifts the guitar
118 - a man walks around the apartment
120 - a man places guitar on the sofa
122 - a man walks around the apartment
125 - C stands up
126 - C walks around the apartment
131 - a man picks a coffee maker jug
132 - a man pours coffee in a cup
134 - C stares at a man
140 - C stares at the window
142 - C stares at a man
144 - a man drinks coffee
145 - C looks around the apartment
150 - C walks around the apartment
152 - C talks to a man
156 - C stares at a bench
158 - C looks around the apartment
164 - C stares at a man
164 - a man talks to C
167 - C looks around the apartment
171 - C follows a man
174 - a man points at the window
175 - C looks around the bedroom
Question 1: What can you deduce about the relationship between C and the man, based on their interactions and behaviors throughout the video?
Question 2: Identify the primary activity in the apartment and discuss its importance/significance. How does it influence the actions and atmosphere of the video?
Question 3: Observe the dynamics of the video in terms of changes in characters' actions, interactions, and spatial movement. How do these shifts contribute to the overall narrative?
5.1.2 Answer prompt
I want you to act as a teacher in the class called "Long-term video understanding." I will provide video action narrations and their timestamps and three highly difficult and diverse questions for your students about the high-level details in the video. I want you to test students' following abilities:
Ability 1: Students’ ability to summarize and compare long parts of the video
I want you to create a difficult multiple-choice exam that tests above student abilities based on the three questions I just provided. Each question should have five similar open-ended but short answers, but only one should be correct. Make it very difficult for students to find the correct answer among all the wrong answers. All answers should be closely related to what happens in the video. Make wrong answers significantly longer than correct answers. Ensure all of the correct answers compress information from narrations them into a concise conclusion. Your answers should not mention any particular timestamps or narrations.
Do not use letters for the answer choices
Print each correct answer exactly as "Correct answer: [full answer]"
Please print each wrong answer on a new line and print each wrong answer as "Wrong answer 1,2,3,4: [full answer]"
O - C stares at the lamp
1 - C looks around the apartment
3 - C talks to a man
6 - C walks around the apartment
10 - C talks to a man
11 - C looks around the apartment
14 - a man plays the guitar
20 - a man walks around the apartment
21 - a man plays the guitar
21 - C walks around the apartment
22 - C stares at the window
24 - C looks around the apartment
27 - C talks to a man
29 - C picks chips from the table
32 - C looks around the apartment
36 - C stares at a man
36 - a man plays the guitar
37 - C walks around the apartment
41 - C sits on the sofa
43 - C stares at a man
44 - a man plays the guitar
49 - a man climbs up the sofa
50 - a man plays the guitar
62 - a man climbs down the sofa
63 - a man walks around the apartment
64 - a man plays the guitar
Questions:
Question: what can you deduce about the relationship between c and the man, based on their interactions and behaviors throughout the video?
Question: identify the primary activity in the apartment and discuss its importance/ significance. how does it influence the actions and atmosphere of the video?
Question: Observe the dynamics of the video in terms of changes in characters' actions, interactions, and spatial movement. How do these shifts contribute to the overall narrative?
Correct answer: They have a casual, friendly relationship.
Wrong answer 1: C and the man are complete strangers and never interacted before.
Wrong answer 2: C is deeply in love with the man, and the man reciprocates those feelings.
Wrong answer 3: They are business associates discussing a financial transaction.
Wrong answer 4: C is an intruder in the man's apartment, and the man is trying to get C to leave.
Correct answer: The primary activity is the man playing the guitar, which sets a relaxed environment.
Wrong answer 1: The main activity is C staring at different objects, indicating a deep curiosity.
Wrong answer 2: The primary activity is a heated argument between C and the man that escalates over time.
Wrong answer 3: The main activity is the man anxiously pacing around the apartment, creating a tense atmosphere.
Wrong answer 4: The primary activity is C and the man planning to commit a crime together.
Correct Answer: Actions and interactions are casual and relaxed, reflecting a comfortable environment.
Wrong Answer A : The video displays a profound sense of conflict and tension arising between the characters.
Wrong Answer B : The man is showing C the issues that need fixing in the apartment in a professional manner.
Wrong Answer C : Both the characters display an increasingly urgent need to solve an issue in the apartment.
Wrong Answer D : C and the man admire and interact with several objects in the apartment that look beautiful.
5.2 Set B
5.2.1 Question and answer prompt
Ability 2: Students' ability to compress information from the video rather than just listing the actions that happened in the video.
Ability 3: Students' ability to identify the most important parts of the video.
Your questions should not mention any particular timestamps or narrations. Remember to make sure the correct answers to your questions do not list information from the narrations but compress them in a concise conclusion.
Examples of good and difficult questions:
"What is the main action of the video?"
"Why did C do action ...?"
AVOID the following types of questions:
"When . . .?"
"How many . . .?"
"How much . . .?"
When announcing the question please label each question as "Question 1,2,3: [full question]"
Timestamps and narrations:
3 - C holds the cloth in his right hand.
5 - the woman picks a carton from the grocery bag on the floor with her right hand.
6 - the woman drops the carton in a cabinet with her left hand.
7 - the woman dips both hands into the grocery bag.
9 - the woman drops a green carton on the floor with her right hand.
12 - C drops the green carton in the cabinet with his right hand.
13 - the woman holds a pack bag in her right hand.
16 - C opens a kitchen cabinet with his left hand.
18 - C removes a cereal pack from the kitchen cabinet with his left hand.
19 - C puts the green carton into the kitchen cabinet with his right hand.
21 - C closes the kitchen cabinet with his left hand.
24 - the woman removes a plastic from the grocery bag with her right hand.
25 - the woman drops the plastic on the floor with her right hand.
33 - C closes a wardrobe with his left hand.
38 - the woman puts a pack into the cabinet with her right hand.
43 - a dog lies down on a bed.
54 - C picks a cloth from the floor with his right hand.
58 - C adjusts the cloth with both hands.
66 - C hangs the cloth on the wall with both hands.
74 - the woman holds a grocery bag in her right hand.
82 - the woman touches her hair with her right hand.
92 - the woman talks with C.
99 - C holds two pieces of cloths in both hands.
100 - the woman picks piece of clothes from a bag with both hands.
100 - C adjusts a camera on his head with his right hand.
103 - C drops the two piece of cloths on a couch with his left hand.
109 - C opens a door with his right hand.
110 - C walks into a toilet.
114 - C holds a red towel in his right hand.
116 - the woman talks with C.
118 - the woman scratches her back with her right hand.
118 - the dog walks into the toilet.
120 - C hangs the red towel on the wall with his right hand.
121 - C holds a towel in his left hand.
129 - the woman holds the towel in her left hand.
129 - the dog walks out of the toilet.
132 - C holds the door with his left hand.
133 - the woman touches her face with her right hand.
148 - C adjusts the camera on his head with his left hand.
149 - C touches his face with his right hand.
150 - the woman holds a bin with both hands.
160 - C opens the door with his right hand.
161 - C walks into the toilet.
163 - the woman walks into the toilet.
166 - the woman drops the bin on the floor with both hands.
168 - the woman adjusts the bin on the floor with both hands.
Question 1: Can you provide a concise summary of the primary activities taking place in the video, highlighting similarities between the actions of C and the woman?
Question 2: Based on the video, what task seems to be the central focus of both C and the woman's actions, and how do they contribute to the accomplishment of this task?
Question 3: Identify the three most important segments of the video and explain the significance of each in the overall context of the actions taking place.
Students are done with the questions. It is time to reveal the correct answers. Correctly provide short one-sentence answers to the questions you created, and additionally generate 4 very similar but wrong answers. Make sure that it will be hard to find the correct answer when combined with the wrong ones. Also, ensure that all the answers are concise and very close to an actual answer in length. Mainly make sure your correct answers do not list information from the narrations but compress them in a concise conclusion.
Question 1:
Correct answer: C and the woman both interact with various objects to organize the space, including placing grocery items and hanging cloths.
Wrong answer 1: [Output disregarded]
Wrong answer 2: [Output disregarded]
Wrong answer 3: [Output disregarded]
Wrong answer 4: [Output disregarded]
Question 2:
Correct answer: The central focus is organizing and tidying up the space, with C and the woman putting items in their respective places.
Question 3:
Correct answer: Key segments include organizing groceries, adjusting household items (e.g., cloths, bins), and interacting with a dog.
5.2.2 Wrong answer prompt
I want you to act as a teacher in the class called "Long-term video understanding." I will provide video action narrations and their timestamps and three highly difficult and diverse questions for your students about the high-level details in the video. I will also show the correct answers to the questions. I want you to test students' following abilities:
166 - the woman drops the bin on the floor with both hands.
168 - the woman adjusts the bin on the floor with both hands.
Question: Can you provide a concise summary of the primary activities taking place in the video, highlighting similarities between the actions of c and the woman?
Correct answer: c and the woman both interact with various objects to organize the space, including placing grocery items and hanging cloths.
Question: Based on the video, what task seems to be the central focus of both c and the woman's actions, and how do they contribute to the accomplishment of this task?
Correct answer: the central focus is organizing and tidying up the space, with c and the woman putting items in their respective places.
Question: Identify the three most important segments of the video and explain the significance of each in the overall context of the actions taking place?
Correct answer: key segments include organizing groceries, adjusting household items (e.g., cloths, bins), and interacting with a dog.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Question 1: Can you provide a concise summary of the primary activities taking place in the video, highlighting similarities between the actions of c and the woman?
Wrong answer 1: C and the woman engage in lengthy conversations about their day while occasionally placing groceries in cabinets and adjusting cloths on walls.
Wrong answer 2: C and the woman focus on preparing an elaborate dinner, flipping through recipe books and chopping vegetables.
Wrong answer 3: Both C and the woman predominantly pay attention to the dog's wellbeing, providing it with food and water and playing with it.
Wrong answer 4: The video revolves around C and the woman working on a DIY project, with tools scattered around the room.
Question 2: Based on the video, what task seems to be the central focus of both c and the woman's actions, and how do they contribute to the accomplishment of this task?
Wrong answer 1: C and the woman are preoccupied with an ongoing debate, their actions aimlessly directed towards unrelated tasks.
Wrong answer 2: They are both primarily absorbed in their individual tasks, occasionally interacting with each other but with no clear common goal.
Wrong answer 3: The central focus of their actions is providing care and attention to the dog, and their actions revolve around its needs.
Wrong answer 4: The video primarily depicts a disagreement between C and the woman, with actions centered around this conflict.
Question 3: Identify the three most important segments of the video and explain the significance of each in the overall context of the actions taking place?
Wrong answer 1: The main segments involve discussing a movie, checking on an overheating stove, and attempting to catch a rodent in the room.
Wrong answer 2: The most important segments consist of C and the woman eating a meal, participating in a board game, and reading books.
Wrong answer 3: Three primary segments include C and the woman debating household chores, searching for lost items, and tending to a crying baby.
Wrong answer 4: Key segments include discussing an upcoming event, taking turns answering a phone call, and checking updates from an ongoing sports game.
6 Our clip length and narration density choice
Figure 8: Heatmap of a number of viable clips over a range of clip length and narration density. There are only a few viable options that offer some degree of balance between the number of clips and the number of narration in the clip. One potential selection is to utilize 3-minute clips with a density of 5 narrations per minute, although this choice bears the significant disadvantage of potentially including clips with an insufficient volume of narration data to generate high-quality results. Another possible choice is to use 1-minute clips with a density of 20 narrations per minute, yet this option carries the drawback of the clips being too brief for the dataset to be very long-term. Hence, we choose 3-minute clips with a narration density of 10 narrations per minute as it offers a satisfactory balance between the number of narrations and clip length for generating EgoSchema.
7 Human curation
Our research utilized the services of a third-party company (not MTurk), for specifically training annotators to ensure quality. The process involved two distinct annotation procedures: data curation and human accuracy testing.
7.1 Curation
Generated data curation was performed by Quantigo employees. These curators were responsible for ensuring that the released EgoSchema dataset is tge highest high quality possible. Here is the exact instructions that was provided to annotators:
The annotation we need is to say that the Question-correct answer-wrong answer set (the whole set) is good if all these three conditions pass:
Condition A) Question is Answerable: The question can be answered from the video and requires more than just a second of video to answer (so, if the answer is not present in the video or, if the answer can be formed with just a few frames (less than say, a second) then it fails this condition).
(Condition B) The Marked Correct Answer is correct: The ""correct answer"" is more the correct answer to the question
(Condition C) The Marked Wrong Answers are wrong: All 4 ""wrong answers"" are less correct than the ""correct answer"" (So for example, if a wrong answer is not completely false, but simply does not contain all the information that the "" correct answer"" does, then it is still a fine ""wrong answer"") IF even one of the marked answer is correct, the set should be labeled as bad.
Condition D) The question is actually long-term: This is a very very important condition. We want the certificate for the question to be at least 30 seconds minimum. If the certificate is non-contiguous (ie. 5 seconds at one place, 20 seconds at another, and 15 more seconds at a third place) the sum of lengths of all the sub-certificates together should be more than 30 seconds. Another example is, if a question can be answered simply from a few frames of the video, the certificate is small (and less than 30 seconds) and hence would fail this condition. Additional detials on how to handle certificate edge cases are provided in the annotator training through examples.
(Condition E) Avoid Boring Questions: Questions that ask about the frequency of something ("How many times..") fail this condition.
If any of these five conditions fail we want the whole set (Question / Correct Answer / Wrong Answer) marked bad.
Optional:
Since GOOD sets are so rare, in cases where it seems that a set is good but a small part of the above five conditions is not being met or, if one/two words were different this can be a good set, please label as MAYBE and we will fix it in the second round. We expect, Good/Bad to be about  $97\%$  of data and Maybe to be not more than  $3\%$ .
Extended notes:
1. In our experience, the wrong answers are made such that they differ from the correct answer in small but crucially meaningful ways. There are many cases where a careless reading of the wrong answer might make it seem that it is correct but upon careful inspection, it will become clear that something about the wrong answer indeed makes it wrong. While this is common, there are indeed cases where the wrong answer is also just as correct as the correct answer. In such cases, the wrong answer fails condition C, and the set is bad.