SlowGuess/NeurIPS_2023_OCR huggingface dataset:
c) Who funded the creation of the dataset? If there is an associated grant, please provide the name of the grantor and the grant name and number.
This project is funded by Meta.
a) What do the instances that comprise the dataset represent (e.g., documents, photos, people, countries)? Are there multiple types of instances (e.g., movies, users, and ratings; people and interactions between them; nodes and edges)? Please provide a description.
Each instance is a set of activity annotations on a particular video in Ego4D [22], organized in a hierarchical form. We show an example annotation in Listing 1.
b) How many instances are there in total (of each type, if appropriate)?
Ego4D Goal-Step is comprised of two main components.
- The goal annotation set consists of 7,353 videos, totaling 2,807 hours. This set covers  $72\%$  of Ego4D after discarding non-procedural or uninteresting videos, and is intended to provide a large-scale training and evaluation dataset for goal inference.
- The step annotation set focuses on the cooking scenario portion of Ego4D, and is intended specifically for procedural activity recognition. It consists of 47,721 densely labeled segments, amounting to 430 hours in total.
Listing 1: Example annotation in a JSON format
{   
"9b58e3ab-7b6d-4e79-9eea-c21420b0ecdc":{ "start_time":0.0210286458333333, "end_time":510.1876953125, "goal_category":"COOKING:MAKE_OMELET", "goal_description":"Make omelette", "goal.wikihow_url":"https://www.wikihow.com/Cook-a-Basic-Omelette ", "summary":[ "Toasting bread on a pan", "Making omelet", "Serving omelet with ketchup"] , "is_procedural":true, "segments":[   
{ "start_time":0, "end_time":56.99209, "step_category":"General cooking activity:Toast bread" "step_description":"Toast bread", "is Continued":false, "is_procedural":true, "is Relevant":"essential", "summary":[ "heat skillet", "toast bread", "trash kitchen waste"
]， "segments": [ { "start_time": 0, "end_time": 13.135, "step_category": "Cook on a stovetop: Turn on the stovetop" "step_description": "preheat the stove-top", "is Continued": false, "is Relevant": "essential", "is_procedural": true, "summary": [ "turn on stove", "preheat the stove-top" ], "segments": [], ... ] ... }
c) Does the dataset contain all possible instances or is it a sample (not necessarily random) of instances from a larger set? If the dataset is a sample, then what is the larger set? Is the sample representative of the larger set (e.g., geographic coverage)? If so, please describe how this representativeness was validated/verified. If it is not representative of the larger set, please describe why not (e.g., to cover a more diverse range of instances, because instances were withheld or unavailable).
The goal annotation set covers  $72\%$  of Ego4D. We filtered out videos based on scenarios that are deemed non-procedural or uninteresting. The discarded scenarios are:
- Eating: "Eating", "Eating at a restaurant", "Eating in a canteen", "Eating in hawker center"
- Talking: "Talking with family members", "Talking with friends/housemates", "Talking to colleagues", "Talking on the phone"
- Attending a meeting: 'Attending a TA session', 'Participating in a meeting'
- Watching something: "Watching tv", "Working at desk", "Reading books", "Video call", "Playing games/video games", "On a screen (phone/laptop)", "Play with cellphone"
- Commuting / moving around: "Bike", "Cycling/jogging", "Car-commuting, road trip", "Skateboard/scooter", "Walking the dog/pet", "Walking on street", "Clothes, other shopping"
The step annotation set contains only cooking scenario videos due to their strong procedural characteristics.
Ego4D metadata includes information about collection sites (university names), which contains geographical information. This allows us to analyze the geographical diversity of the goal and step annotation sets. The distribution of collection sites are (see Figure 6):
- Ego4D: 'cmu': 1898, 'unict': 1457, 'iiith': 1233, 'minnesota': 1123, 'kaust': 960, 'frl_track_1_public': 836, 'utokyo': 780, 'bristol': 778, 'cmu_africa': 144, 'uniandes': 137, 'nus': 115, 'indiana': 95, 'georgiatech': 55.
- Goal annotation set: 'cmu': 1675, 'unict': 1214, 'utokyo': 840, 'iiith': 832, 'minnesota': 701, 'frl_track_1_public': 606, 'bristol': 529, 'kaust': 524, 'cmu_africa': 144, 'nus': 115, 'uniandes': 74, 'indiana': 62, 'georgiatech': 37.
- Step annotation set: 'utokyo': 237, 'iiith': 204, 'bristol': 101, 'minnesota': 96, 'unict': 77, 'kaust': 77, 'cmu_africa': 41, 'nus': 7.
We can see the goal annotation set closely follows the distribution of the entire Ego4D dataset. The step annotation set is different from those distributions. We note that this is because cooking
scenario videos were collected only in those collection sites, rather than us intentionally selecting videos from certain collection sites.
d) What data does each instance consist of? "Raw" data (e.g., unprocessed text or images) or features? In either case, please provide a description.
Each instance consists of time interval, goal/step category and description, text summary, and various other auxiliary information. For full details of the information contained in each instance, please see Listing 1.
e) Is there a label or target associated with each instance? If so, please provide a description. There are various labels associated with each instance. Below shows the definition of each field.
- start time and end_time: Time interval
- goal_category and step_category: goal and step category label
- goal_description and step_description: goal and step description in natural language
- goal.wikihow_url: A wikiHow URL that best describes the activity captured in a video
- summary: A bullet-pointed summary of steps contained in each segment
- segments: A list of step segments (or substep segments) under a given goal (or step) segment
- is_procedural: A boolean flag indicating whether this segment contains procedural activity.
- is Continued: A boolean flag indicating whether this segment is a continuation of an activity (goal, step, or substep) from the most recent earlier segment of the same activity.
- is Relevant: One of "essential", "optional", "irrelevant" indicating how relevant this segment is to its parent segment.
f) Is any information missing from individual instances? If so, please provide a description, explaining why this information is missing (e.g., because it was unavailable). This does not include intentionally removed information, but might include, e.g., redacted text.
Everything that the annotators have provided is included. No data is missing.
g) Are relationships between individual instances made explicit (e.g., users' movie ratings, social network links)? If so, please describe how these relationships are made explicit.
Each individual video in Ego4D contains metadata about collection site (university name), from which relationships can be established across individual instances. This information is available in the Ego4D dataset. In Ego4D Goal-Step, to avoid redundancy, we do not explicitly provide the collection site information, but this can be easily retrieved by using the unique video identifier (the keys in the JSON dictionary, e.g., "9b58e3ab-7b6d-4e79-9eea-c21420b0ecd" in Listing 1).
h) Are there recommended data splits (e.g., training, development/Validation, testing)? If so, please provide a description of these splits, explaining the rationale behind them.
We provide data splits for training, validation, and testing purposes. We release full annotations for the training and validation splits while withholding the test split. To facilitate evaluation, we will set up and maintain a test server on EvalAI. Participants can upload their results to the server, where they will be evaluated automatically.
We divide the data into training  $(70\%)$ , validation  $(15\%)$ , and test  $(15\%)$  splits. The split is performed at the video level to ensure no information leakage across splits. To achieve this, we employed stratified sampling over videos using the train_test_split function from the sklearn Python package. Each video is assigned a single category label. We adopted a greedy assignment approach, iterating over step categories sorted by their frequency. Step labels are assigned to videos until a minimum number is reached, which we set at a minimum of 4 instances per step category.
i) Are there any errors, sources of noise, or redundancies in the dataset? If so, please provide a description.
As described in Section 3.2 of the main paper, annotators provide both category labels and natural language descriptions for segments of goals, steps, and substeps identified in a video. When annotators cannot find a category label in the taxonomy, they select "other" and suggest a new category in the description. We periodically review those descriptions labeled with "other" and either map them to existing categories or add them to the taxonomy. This is done through keyword mapping and manual verification. Due to the manual mapping process and the inherent complexity of human activity, it is possible that the process is noisy.
j) Is the dataset self-contained, or does it link to or otherwise rely on external resources (e.g., websites, tweets, other datasets)? If it links to or relies on external resources, a) are there guarantees that they will exist, and remain constant, over time; b) are there official archival versions of the complete dataset (i.e., including the external resources as they existed at the time the dataset was created); c) are there any restrictions (e.g., licenses, fees) associated with any of the external resources that might apply to a dataset consumer? Please provide descriptions of all external resources and any restrictions associated with them, as well as links or other access points, as appropriate.
Ego4D Goal-Step is a new set of annotations on existing videos in Ego4D, and will be hosted directly on the official Ego4D repository alongside other annotations.
The repository itself is supported by Meta and the 15 university Ego4D consortium, and the board and consortium exist independent of Meta. There is funded work already underway for future versions of the dataset and no doubts about the long-term persistence of the consortium or the flagship video dataset itself which these annotations are based upon.
Each version of the dataset that is updated is marked as an individual update, and prior versions are always available in their original form via the CLI. The dataset will also include appropriate metadata to confirm the correct versions and videos are downloaded and used beyond that guarantee.
Ego4D Goal-Step annotations are available via the CLI and do not require a license or any restrictions (once publicly released). For the Ego4D videos themselves, there is no cost, but an individual or entity must sign the license and receive approval to download the access keys as detailed above. Requests are only rejected for incorrect submissions or from being from a country with comprehensive US, UK or EU trade restrictions (currently Crimea, Donetsk, and Luhansk regions of Ukraine, Russia, Cuba, North Korea, Iran, and Syria - though that is subject to change).
k) Does the dataset contain data that might be considered confidential (e.g., data that is protected by legal privilege or by doctor-patient confidentiality, data that includes the content of individuals' non- public communications)? If so, please provide a description.
No information contained in Ego4D Goal-Step is considered confidential.
1) Does the dataset contain data that, if viewed directly, might be offensive, insulting, threatening, or might otherwise cause anxiety? If so, please describe why.
m) Does the dataset identify any subpopulations (e.g., by age, gender)? If so, please describe how these subpopulations are identified and provide a description of their respective distributions within the dataset.
No explicit process was in place to identify any subpopulations during the annotation process. The activity labels were indiscriminate of the identity of the camera wearer who recorded the videos.
n) Is it possible to identify individuals (i.e., one or more natural persons), either directly or indirectly (i.e., in combination with other data) from the dataset? If so, please describe how.
It is not possible to identify individuals from the labels provided in Ego4D Goal-Step. All videos in Ego4D went through a set of de-identification processes to ensure a high standard of data privacy, see Appendix B of Grauman et al. [22] for details.
o) Does the dataset contain data that might be considered sensitive in any way (e.g., data that reveals race or ethnic origins, sexual orientations, religious beliefs, political opinions or union memberships, or locations; financial or health data; biometric or genetic data; forms of government identification, such as social security numbers; criminal history)? If so, please provide a description.
a) How was the data associated with each instance acquired? Was the data directly observable (e.g., raw text, movie ratings), reported by subjects (e.g., survey responses), or indirectly inferred/derived from other data (e.g., part-of-speech tags, model-based guesses for age or language)? If the data was
reported by subjects or indirectly inferred/derived from other data, was the data validated/verified? If so, please describe how.
The videos were directly observable by the annotators, through a specialized tool with a video player and input fields tailored for our annotation (see the annotation user interface in Figure 7).
b) What mechanisms or procedures were used to collect the data (e.g., hardware apparatuses or sensors, manual human curation, software programs, software APIs)? How were these mechanisms or procedures validated?
We used an internal tool developed at Meta (Figure 7) for web-based annotation. The same tool was used for the creation of the Ego4D dataset, validating its robustness and scalability.
c) If the dataset is a sample from a larger set, what was the sampling strategy (e.g., deterministic, probabilistic with specific sampling probabilities)?
See our response in D.2.c.
d) Who was involved in the data collection process (e.g., students, crowdworkers, contractors) and how were they compensated (e.g., how much were crowdworkers paid)?
We contracted a third party vendor to manage the annotation process, and checked privacy and ethical compliance through rigorous reviews. The amount of compensation were determined via contract, and the payments were made on a monthly basis. Because of multiple annotation projects involved with the vendor, the exact breakdown for how much they were paid is unknown to us.
e) Over what timeframe was the data collected? Does this timeframe match the creation timeframe of the data associated with the instances (e.g., recent crawl of old news articles)? If not, please describe the timeframe in which the data associated with the instances was created.
The entire annotation took 6 months to collect, from late 2022 to mid 2023. The annotation timeframe does not match the creation timeframe of video recordings, which was made in year 2021 through 2022.
f) Were any ethical review processes conducted (e.g., by an institutional review board)? If so, please provide a description of these review processes, including the outcomes, as well as a link or other access point to any supporting documentation.
Yes. This annotation project went through a rigorous internal review process at Meta for privacy and ethical compliance.
g) Did you collect the data from the individuals in question directly, or obtain it via third parties or other sources (e.g., websites)?
Collection or recording of new videos was not as part of Ego4D Goal-Step annotation. The annotations were provided through a third party vendor who managed contracted workers. The vendor managed all contractors, applying their well-established processes for annotator recruitment, training, auditing, and management. The contractors use our web-based annotation tool (Figure 7) to annotate videos.
h) Were the individuals in question notified about the data collection? If so, please describe (or show with screenshots or other information) how notice was provided, and provide a link or other access point to, or otherwise reproduce, the exact language of the notification itself.
Yes, our annotators are contracted workers who fully comprehend the nature of their tasks. They have been duly informed that we are gathering annotations from them, and these annotations will be made available to the public.
i) Did the individuals in question consent to the collection and use of their data? If so, please describe (or show with screenshots or other information) how consent was requested and provided, and provide a link or other access point to, or otherwise reproduce, the exact language to which the individuals consented.
j) If consent was obtained, were the consenting individuals provided with a mechanism to revoke their consent in the future or for certain uses? If so, please provide a description, as well as a link or other access point to the mechanism (if appropriate).
Figure 6: Comparison of geographical distributions of videos on three different sets: the original Ego4D [22], our goal annotation set, and our step annotation set. The goal annotation set follows the original distribution, while the step annotation set differs because of its focuses cooking scenarios.
Figure 7: The user interface used for data annotation.
k) Has an analysis of the potential impact of the dataset and its use on data subjects (e.g., a data protection impact analysis) been conducted? If so, please provide a description of this analysis, including the outcomes, as well as a link or other access point to any supporting documentation.
C.4 Processing, cleaning, labeling
a) Was any preprocessing/cleaning/labeling of the data done (e.g., discretization or bucketing, tokenization, part-of-speech tagging, SIFT feature extraction, removal of instances, processing of missing values)? If so, please provide a description. If not, you may skip the remaining questions in this section.
We mapped natural language descriptions of goals, steps, and substeps to their corresponding categories via keyword mapping followed by manual verification; see D.2.i and Section 3.2 of the main paper for details.
b) Was the "raw" data saved in addition to the preprocessed/cleaned/ labeled data (e.g., to support unanticipated future uses)? If so, please provide a link or other access point to the "raw" data.
Yes. We also release the "raw" natural language descriptions in in Ego4D Goal-Step; see 'goal_description' and 'step_description in Listing 1
c) Is the software that was used to preprocess/clean/label the data available? If so, please provide a link or other access point.
a) Has the dataset been used for any tasks already? If so, please provide a description.
Ego4D Goal-Step is not public yet, so no other papers have used it for their tasks. However, our main paper demonstrates how it can be used in three scenarios: temporal goal/step localization, online goal/step detection, and step grounding. See Section 4 for details.
b) Is there a repository that links to any or all papers or systems that use the dataset? If so, please provide a link or other access point.
c) What (other) tasks could the dataset be used for?
Besides the tasks we demonstrate in the paper, Ego4D Goal-Step can support other tasks in procedural video understanding, such as hierarchical action segmentation [17], action anticipation [18], procedure planning [9], and task graph learning [31], and video summarization [38].
d) Is there anything about the composition of the dataset or the way it was collected and preprocessed/cleaned/labeled that might impact future uses? For example, is there anything that a dataset consumer might need to know to avoid uses that could result in unfair treatment of individuals or groups (e.g., stereotyping, quality of service issues) or other risks or harms (e.g., legal risks, financial harms)? If so, please provide a description. Is there anything a dataset consumer could do to mitigate these risks or harms?
e) Are there tasks for which the dataset should not be used? If so, please provide a description.
We acknowledge that Ego4D Goal-Step is intended for research purposes and should not be regarded as a comprehensive dataset encompassing the full range of daily human activities. Similar to other large-scale real-world datasets, Ego4D Goal-Step exhibits skewed distribution of activities, subjects, and settings, despite the best effort in Ego4D to capture real-world authenticity and comprehensive diversity. As such, models trained on our dataset may exhibit biases towards the specific activities included in the dataset, resulting in a limited coverage of our everyday living scenarios.
a) Will the dataset be distributed to third parties outside of the entity (e.g., company, institution, organization) on behalf of which the dataset was created? If so, please provide a description.
Yes. The dataset will be publicly available on the internet.
b) How will the dataset will be distributed (e.g., tarball on website, API, GitHub)? Does the dataset have a digital object identifier (DOI)?
Ego4D Goal-Step will be distributed through the official Ego4D download protocol, which can be found in https://ego4d-data.org. There will be no DOI assigned specifically for Ego4D Goal-Step.
c) When will the dataset be distributed?
We will release Ego4D Goal-Step around before December 2023.
d) Will the dataset be distributed under a copyright or other intellectual property (IP) license, and/or under applicable terms of use (ToU)? If so, please describe this license and/or ToU, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms or ToU, as well as any fees associated with these restrictions.
Ego4D Goal-Step will be distributed under the standard Ego4D license agreement, which can be found in https://ego4ddataset.com/.
e) Have any third parties imposed IP-based or other restrictions on the data associated with the instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any relevant licensing terms, as well as any fees associated with these restrictions.
f) Do any export controls or other regulatory restrictions apply to the dataset or to individual instances? If so, please describe these restrictions, and provide a link or other access point to, or otherwise reproduce, any supporting documentation.
a) Who will be supporting/hosting/maintaining the dataset?
The Ego4D consortium will support/host/maintain it.
b) How can the owner/curator/manager of the dataset be contacted (e.g., email address)?
The primary contact is Yale Song (yalesong@meta.com).
c) Is there an erratum? If so, please provide a link or other access point.
Not so far, it has not been released yet.
d) Will the dataset be updated (e.g., to correct labeling errors, add new instances, delete instances)? If so, please describe how often, by whom, and how updates will be communicated to dataset consumers (e.g., mailing list, GitHub)?
Yes, we plan to update the dataset with labeling corrections and newly annotated labels in the future. The updates will be made on the official Ego4D website, https://ego4d-data.org/docs/updates/.
e) If the dataset relates to people, are there applicable limits on the retention of the data associated with the instances (e.g., were the individuals in question told that their data would be retained for a fixed period of time and then deleted)? If so, please describe these limits and explain how they will be enforced.
f) Will older versions of the dataset continue to be supported/hosted/maintained? If so, please describe how. If not, please describe how its obsolescence will be communicated to dataset consumers.
Yes, older versions will continue to be supported/hosted/maintained through the Ego4D repository.