MOMA: Multi-Object Multi-Actor Activity Parsing

∗These authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . While human activity recognition has advanced considerably in recent years , the datasets [ 8 , 14 , 68 ] and models [ 69 , 74 , 17 ] that underlie these improvements still treat human activities as monolithic events . Typically , the action classification task is only conducted on a coarse temporal scale and on a limited set of activity labels . In contrast , real-world activities in domains such as healthcare [ 87 , 47 , 53 ] , surveillance [ 52 , 78 , 29 ] , and entertainment [ 77 , 57 , 70 ] are complex , which involve intricate social interactions between humans utilizing different objects and sequences of steps they perform to achieve a goal . The understanding of these activities requires more fine-grained and structured reasoning beyond single semantic labels . Recently , efforts have been made in developing activity recognition models to address this disparity , including temporal localization [ 6 , 95 ] , hierarchical reasoning [ 65 ] , atomic action reasoning [ 22 , 21 ] , and action decomposition [ 27 ] . However , none of them fully addresses the needs in real-world applications . We argue that a complex activity recognition system invokes understanding the detailed hierarchy and composition of activity components in a video sequence [ 89 ] . Firstly , human activities are by nature hierarchical [ 3 ] and abstract behaviors ( such as dining service ) are composed of a series of low-level body movements and interactions ( such as speaking , sitting , and holding ) . Defining a vocabulary of activity labels can be ambiguous if multiple levels of granularity ( i.e. , a hierarchy ) are not adequately defined . Secondly , when multiple entities exist in the video , understanding the composition of the video by reasoning each actor ’ s social role [ 61 ] , each object ’ s affordance , and their relationships [ 27 ] is crucial to visual dynamics understanding ( see Fig . 1 ) . Towards this goal , we propose a redefined Action Parsing [ 39 , 43 , 66 ] for complex human activity recognition . Our proposed activity parsing task parses complex activities into multiple timescales and spatial-temporal grounding of entities and relationships . The representation consists of four levels of hierarchy : ( 1 ) activity , ( 2 ) sub-activity , ( 3 ) atomic action , and ( 4 ) action hypergraph . More specifically , an action hypergraph is a spatial-temporal graph [ 50 ] tailored for multi-object and multiactor scenarios , in which a hyperedge can encode higher-order relationships by connecting any number of vertices ( i.e. , entities , including actors and objects ) . Hyperedges , which can still trivially represent pairwise relationships , define more comprehensive relationships than the pairwise relations prevalent in the traditional scene graphs . In this representation , the contextual information in high-level activities and sub-activities facilitates the fine-grained recognition of scene dynamics on the atomic action and action hypergraph levels ; low-level action hypergraphs and their constituents provide a grounding for high-level activity recognition necessary for its interpretability and transferability . Furthermore , we propose a novel model , termed Hypergraph Activity Parsing ( HGAP ) network , for the activity parsing task on videos . We also explore other tasks such as video classification and role classification with HGAP . We compare HGAP with an existing graph neural network approach [ 83 ] and the state-of-the-art activity recognition model [ 16 ] . By learning with Action Hypergraph , HGAP achieves comparable results with 3D CNN models without pre-training on videos from the same domain , and HGAP with X3D backbone can achieve superior results for activity parsing . Lastly , we introduce the MOMA ( Multi-Object , Multi-Actor ) dataset , consisting of 17 activity categories , 67 sub-activity categories , 52 atomic action categories , 120 object categories , 20 actor categories , and 75 relationship ( static and dynamic ) categories . The MOMA dataset has three unique features : ( 1 ) activity annotations with four levels of granularity , which enable activity parsing on different levels of abstraction ; ( 2 ) multi-object , multi-actor , and categorical labels for actors ( i.e. , social roles ) and objects , which provide exhaustive details of the associated activity ; ( 3 ) higher-order relationships [ 12 , 1 ] between entities , which expressively capture the relationships in crowded and complex scenes . To the best of our knowledge , MOMA is the first dataset with these features . 2 Related work . Action recognition refers to the problem of recognizing human actions in videos , with two major tasks in the domain ( 1 ) action classification [ 91 ] and ( 2 ) temporal action detection ( TAD ) [ 79 ] . For ( 1 ) , recent advances are toward 3D CNNs [ 8 , 23 , 60 , 73 , 80 ] such as C3D [ 73 ] , I3D [ 8 ] , SlowFast [ 17 ] , and X3D [ 16 ] that jointly model spatial and temporal features . ( 2 ) [ 79 ] in comparison is a harder task , and works often follow either a proposal-then-classification paradigm [ 67 , 81 , 19 , 42 , 40 ] or an approach to model the two steps at same time [ 41 , 5 , 88 , 2 ] . Today , researchers have pushed human action understanding to a deeper level in two major directions that our work is based upon : ( 1 ) action as hierarchical representations , ( 2 ) action as compositions of visual relationships . To jointly model these two representation , we identify graph neural network and action parsing as two promising tools for our study , briefly surveyed below . Hierarchical representation learning is often studied on data with a network structure [ 4 , 90 , 9 , 51 ] , yet its application in video action recognition is an emerging direction . Drawing from Cognitive Science studies [ 89 ] , a hierarchy of human actions can either be taxonomic [ 14 , 11 ] ( e.g. , playing basketball is a kind of sport ) , or partonomic [ 33 , 35 , 27 , 65 ] ( e.g. , take an order is a step of dining service ) . [ 45 , 72 ] leveraged taxonomy hierarchy for action prediction , whereas [ 35 , 33 , 34 ] exploited the two-level partonomy of actions for few-shot action classification . This work aims to apply a four-level partonomic hierarchy to solve the task of activity parsing . In particular , we break high-level activity to sub-activity , then atomic action , and lastly a frame-wise representation for visual human-object relationship , as discussed in the next subsection . Visual relationship and compositionality has been studied in video domain to represent human actions . For example , Something-Else [ 49 ] models object relationship as tuples , Action Genome [ 27 ] formulates human-object relationship in video as a spatial-temporal scenegraph , and STAG [ 25 ] views changing relationship as dynamic graph representing an action . To model this , models [ 46 , 82 , 38 ] and new loss functions [ 93 ] are proposed . In recent years , graph neural networks [ 85 , 56 ] have been found to be a promising tool for both generating and learning from visual compositions . Graph neural networks ( GNNs ) were first introduced in [ 20 , 64 ] , and has been recently adopted for video understanding . Lines of work have studied its application on spatial-temporal graph representations such as skeleton data [ 84 , 37 ] or visual relationships [ 59 , 54 ] . More recently , Hypergraph Neural Network [ 18 ] was proposed and greatly inspired our work as we see its potential in improving scene graph representations and deepen model ’ s cognitive understanding to human actions . Action parsing [ 66 , 15 , 43 , 44 , 39 , 58 , 76 , 55 , 75 ] is often referred to as modeling multiple characteristics of actions simultaneously . For example , [ 39 ] jointly models human joints , objects , and action labels in images , whereas [ 44 ] explored predicting human pose , action attributes and action labels together in videos . In this work , we redefine this task to be activity parsing . We will ask the model to predict the activity , to temporally detect the sub-actions that occurred , to localize the people and the objects that involved , and to recognize the visual relationships related to the ongoing activity . 3 Data representation and dataset . The MOMA dataset is structured in a four-level hierarchy based on the activity partonomy , with rich annotations at each level . As shown in Figure 1 , the activity dining service consists of multiple steps ( i.e. , sub-activities ) , including take an order , serve wine , serve food , and clean up the table . Further , the sub-activity taking an order also involves multiple entities . It consists of several actor-centric atomic actions : sit , talk to someone , write on something , etc . At the lowest level , we provide a dense annotation of actor-object relationships . In particular , we annotate higher order relationships , such as ( { waiter } , holding , { pen , sheet } ) , ( { customer , waiter } , on the side of , { dining table } ) . We argue that its outcome , the action hypergraph is more effective when representing multi-object multi-actor interaction . In addition , we provide temporal localization for each sub-level action , and assign a unique ID for each actor/object in a video for temporal tracking . 3.1 Dataset statistics . MOMA contains 373 raw videos at activity level , 2,364 trimmed videos at sub-activity level , and 12,057 atomic action instances . This includes 17 activity classes , 67 sub-activity classes , and 52 atomic action classes . At frame level , we provide action hypergraph annotations for 37 , 428 frames , with 164 , 162 actor/object instances of 20 actor classes and 120 object classes , and 119 , 132 relationship instances of 75 relationship classes . On average , there are 4.39 actors/objects and 3.18 higher-order relationships per frame , 5 instances of atomic actions per clip , and 6.34 instances of sub-activities per untrimmed video ( Table 1 ) . Detailed information is provided in the supplementary material about the distributions of activities , subactivities , atomic activities , entities , relationships , and their co-occurrences . We have adopted a series of protocols to mitigate potential ethical issues of the dataset : ( 1 ) videos are sourced from the YouTube platform , where each video has been cross-verified by multiple annotators to avoid offensive contents ; ( 2 ) we do not provide videos , but instead provide YouTube IDs and download script ; ( 3 ) we control personally identifiable data so that only links to publicly available videos are released , and owners of the video can disable the accessibility at any time ; ( 4 ) any release of imaging data for demonstration purpose will be verified by our group to not contain any personally identifiable information , and will involve using face blurring algorithms . 3.2 Partonomic hierarchy . The MOMA Representation features a four-level partonomic representation for complex activities . Activity . An activity [ 69 ] ( e.g. , dining service ) is a high-level description of an event done by people for a specific purpose . Typically , it is an abstraction of several sub-processes carried out by multiple entities . Spurious features , such as background , often provide contextual clues important for sub-optimal activity classification . Sub-activity . A sub-activity is a stage making up part of a larger activity . Typically , the sub-activities ( e.g. , serve food , clean up the table ) from the same activity may take place within the same environment . Hence , the key to segmenting and classifying a sub-activity is to identify actors , objects , and their interactions . Atomic action . An atomic action [ 22 ] represents an actor ’ s body movement or interaction with other entities . At the most primitive level , atomic actions are actor-centric [ 71 ] , but an actor can simultaneously perform multiple atomic actions . Generally speaking , atomic actions must be spatially and temporally localized in order to be recognized . Action hypergraph . An action hypergraph is a spatial-temporal heterogeneous hypergraph that incorporates static and dynamic information about actors and objects . In an action hypergraph , a node represents an entity ( actor or object ) and a hyperedge ( which can join more than two nodes ) represents a higher-order relationship ( static relationship or dynamic relationship ) between the connected entities . As a spatial-temporal graph , an action hypergraph changes with time [ 50 ] and its nodes are spatially localized and tracked as bounding box tracklets on the raw video . As a result , although the action hypergraph is annotated per frame , based on the granularity of our prediction , it is flexible to be formed into a hypergraph that captures information about higher level actions ( e.g . atomic actions , sub-activities , or activities ) .

Summary: This paper introduces Activity Parsing for the understanding of complex human activities based on a hierarchical structure describing human actions at 4 different levels. As an improvement from traditional pairwise entity representations, the Action Hyperedge is used to represent the more complex relationships between multiple humans and objects. Also the Action Hypergraph provides better interpretability for reasoning of high-level actions or activities. Followed by the concept of the hierarchical Activity Parsing, a new, meticulously-annotated video dataset, MOMA, is released and an HGAP network is proposed. In the experiment, the HGAP network achieves comparable results with the state-of-the-arts in several tasks such as video and role classification.