1. Task Decomposition. We first decompose the task to ensure a standardized workflow for execution. For instance, we begin by annotating the finest granularity (action), and subsequently continue with the annotations of activity and story. This approach ensures accuracy and consistency in the fundamental content throughout various levels of granularity. Given that action is the finest granularity and its annotation quality may affect activity and story, we refer to film narrative literature and English grammar materials to further decompose the description of an action. This decomposition involves identifying the tracking target (who), the motion (what), the presence of a third-party object (if applicable), the location (where), and the time interval (when). By obtaining these specific details, annotators can attain a standardized and comprehensive description of the action.
2. Annotator Selection. Considering the difficulty of controlling annotation quality in crowdsourcing, we chose highly cognitive graduate students with experience in dataset annotation to form an annotation team. Team members not only have experience in annotating vision-based datasets represented by VideoCube but also have experience in annotating image datasets in visual psychology. They have a solid foundation in dataset construction in fields such as computer vision and cognitive psychology. All team members undergo standardized training before formal annotation to ensure their understanding of task characteristics and annotation rules. Additionally, the training session includes 10 video examples of different types, requiring annotators to comprehend the annotation process and details.
3. Annotation Workflow. (1) The formal annotation process begins, wherein annotation personnel is grouped based on video types, including cartoons, movies, TV shows, sports, performances, and documentaries. Any issues requiring discussion will be documented, followed by a comprehensive discussion among all personnel, and then the annotation process for that particular sequence will commence. (2) For instance, in the case of a sniper rifle as the target, which term should be employed: "gun" or "sniper rifle"? Through consulting relevant materials and conducting discussions, we have concluded that the fundamental principle of annotation is to incorporate human comprehension into semantic labels. As there is no second firearm in the sequence and the term "gun" encompasses "sniper rifle" semantically, the usage of "gun" aligns with commonplace terminology. Nevertheless, if a second gun appears in the sequence, "sniper rifle" may be employed to underscore the target's distinctiveness. (3) Furthermore, to enhance the standardization of annotations, we refer to WordNet to construct verb and noun lists. Initially, annotators will choose candidate terms from the current vocabulary lists to depict the essential elements of the scene, aiming to maintain consistency in the portrayal of actions across varying sequences to the greatest extent possible. If there are no appropriate terms found in the candidate list, annotators will employ new vocabulary to depict the elements and subsequently incorporate them into the candidate list, supplemented with relevant examples for future annotation reference.
4. Quality Review. After completing the annotation for all sequences, we will review the content to ensure its quality. Additionally, we utilize the Stanford CoreNLP [45], a natural language processing toolkit, to examine the semantic annotations and ensure the grammatical structure's completeness.