MOMA: Multi-Object Multi-Actor Activity Parsing

∗These authors contributed equally . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . 1 Introduction . While human activity recognition has advanced considerably in recent years , the datasets [ 8 , 14 , 68 ] and models [ 69 , 74 , 17 ] that underlie these improvements still treat human activities as monolithic events . Typically , the action classification task is only conducted on a coarse temporal scale and on a limited set of activity labels . In contrast , real-world activities in domains such as healthcare [ 87 , 47 , 53 ] , surveillance [ 52 , 78 , 29 ] , and entertainment [ 77 , 57 , 70 ] are complex , which involve intricate social interactions between humans utilizing different objects and sequences of steps they perform to achieve a goal . The understanding of these activities requires more fine-grained and structured reasoning beyond single semantic labels . Recently , efforts have been made in developing activity recognition models to address this disparity , including temporal localization [ 6 , 95 ] , hierarchical reasoning [ 65 ] , atomic action reasoning [ 22 , 21 ] , and action decomposition [ 27 ] . However , none of them fully addresses the needs in real-world applications . We argue that a complex activity recognition system invokes understanding the detailed hierarchy and composition of activity components in a video sequence [ 89 ] . Firstly , human activities are by nature hierarchical [ 3 ] and abstract behaviors ( such as dining service ) are composed of a series of low-level body movements and interactions ( such as speaking , sitting , and holding ) . Defining a vocabulary of activity labels can be ambiguous if multiple levels of granularity ( i.e. , a hierarchy ) are not adequately defined . Secondly , when multiple entities exist in the video , understanding the composition of the video by reasoning each actor ’ s social role [ 61 ] , each object ’ s affordance , and their relationships [ 27 ] is crucial to visual dynamics understanding ( see Fig . 1 ) . Towards this goal , we propose a redefined Action Parsing [ 39 , 43 , 66 ] for complex human activity recognition . Our proposed activity parsing task parses complex activities into multiple timescales and spatial-temporal grounding of entities and relationships . The representation consists of four levels of hierarchy : ( 1 ) activity , ( 2 ) sub-activity , ( 3 ) atomic action , and ( 4 ) action hypergraph . More specifically , an action hypergraph is a spatial-temporal graph [ 50 ] tailored for multi-object and multiactor scenarios , in which a hyperedge can encode higher-order relationships by connecting any number of vertices ( i.e. , entities , including actors and objects ) . Hyperedges , which can still trivially represent pairwise relationships , define more comprehensive relationships than the pairwise relations prevalent in the traditional scene graphs . In this representation , the contextual information in high-level activities and sub-activities facilitates the fine-grained recognition of scene dynamics on the atomic action and action hypergraph levels ; low-level action hypergraphs and their constituents provide a grounding for high-level activity recognition necessary for its interpretability and transferability . Furthermore , we propose a novel model , termed Hypergraph Activity Parsing ( HGAP ) network , for the activity parsing task on videos . We also explore other tasks such as video classification and role classification with HGAP . We compare HGAP with an existing graph neural network approach [ 83 ] and the state-of-the-art activity recognition model [ 16 ] . By learning with Action Hypergraph , HGAP achieves comparable results with 3D CNN models without pre-training on videos from the same domain , and HGAP with X3D backbone can achieve superior results for activity parsing . Lastly , we introduce the MOMA ( Multi-Object , Multi-Actor ) dataset , consisting of 17 activity categories , 67 sub-activity categories , 52 atomic action categories , 120 object categories , 20 actor categories , and 75 relationship ( static and dynamic ) categories . The MOMA dataset has three unique features : ( 1 ) activity annotations with four levels of granularity , which enable activity parsing on different levels of abstraction ; ( 2 ) multi-object , multi-actor , and categorical labels for actors ( i.e. , social roles ) and objects , which provide exhaustive details of the associated activity ; ( 3 ) higher-order relationships [ 12 , 1 ] between entities , which expressively capture the relationships in crowded and complex scenes . To the best of our knowledge , MOMA is the first dataset with these features . 2 Related work . Action recognition refers to the problem of recognizing human actions in videos , with two major tasks in the domain ( 1 ) action classification [ 91 ] and ( 2 ) temporal action detection ( TAD ) [ 79 ] . For ( 1 ) , recent advances are toward 3D CNNs [ 8 , 23 , 60 , 73 , 80 ] such as C3D [ 73 ] , I3D [ 8 ] , SlowFast [ 17 ] , and X3D [ 16 ] that jointly model spatial and temporal features . ( 2 ) [ 79 ] in comparison is a harder task , and works often follow either a proposal-then-classification paradigm [ 67 , 81 , 19 , 42 , 40 ] or an approach to model the two steps at same time [ 41 , 5 , 88 , 2 ] . Today , researchers have pushed human action understanding to a deeper level in two major directions that our work is based upon : ( 1 ) action as hierarchical representations , ( 2 ) action as compositions of visual relationships . To jointly model these two representation , we identify graph neural network and action parsing as two promising tools for our study , briefly surveyed below . Hierarchical representation learning is often studied on data with a network structure [ 4 , 90 , 9 , 51 ] , yet its application in video action recognition is an emerging direction . Drawing from Cognitive Science studies [ 89 ] , a hierarchy of human actions can either be taxonomic [ 14 , 11 ] ( e.g. , playing basketball is a kind of sport ) , or partonomic [ 33 , 35 , 27 , 65 ] ( e.g. , take an order is a step of dining service ) . [ 45 , 72 ] leveraged taxonomy hierarchy for action prediction , whereas [ 35 , 33 , 34 ] exploited the two-level partonomy of actions for few-shot action classification . This work aims to apply a four-level partonomic hierarchy to solve the task of activity parsing . In particular , we break high-level activity to sub-activity , then atomic action , and lastly a frame-wise representation for visual human-object relationship , as discussed in the next subsection . Visual relationship and compositionality has been studied in video domain to represent human actions . For example , Something-Else [ 49 ] models object relationship as tuples , Action Genome [ 27 ] formulates human-object relationship in video as a spatial-temporal scenegraph , and STAG [ 25 ] views changing relationship as dynamic graph representing an action . To model this , models [ 46 , 82 , 38 ] and new loss functions [ 93 ] are proposed . In recent years , graph neural networks [ 85 , 56 ] have been found to be a promising tool for both generating and learning from visual compositions . Graph neural networks ( GNNs ) were first introduced in [ 20 , 64 ] , and has been recently adopted for video understanding . Lines of work have studied its application on spatial-temporal graph representations such as skeleton data [ 84 , 37 ] or visual relationships [ 59 , 54 ] . More recently , Hypergraph Neural Network [ 18 ] was proposed and greatly inspired our work as we see its potential in improving scene graph representations and deepen model ’ s cognitive understanding to human actions . Action parsing [ 66 , 15 , 43 , 44 , 39 , 58 , 76 , 55 , 75 ] is often referred to as modeling multiple characteristics of actions simultaneously . For example , [ 39 ] jointly models human joints , objects , and action labels in images , whereas [ 44 ] explored predicting human pose , action attributes and action labels together in videos . In this work , we redefine this task to be activity parsing . We will ask the model to predict the activity , to temporally detect the sub-actions that occurred , to localize the people and the objects that involved , and to recognize the visual relationships related to the ongoing activity . 3 Data representation and dataset . The MOMA dataset is structured in a four-level hierarchy based on the activity partonomy , with rich annotations at each level . As shown in Figure 1 , the activity dining service consists of multiple steps ( i.e. , sub-activities ) , including take an order , serve wine , serve food , and clean up the table . Further , the sub-activity taking an order also involves multiple entities . It consists of several actor-centric atomic actions : sit , talk to someone , write on something , etc . At the lowest level , we provide a dense annotation of actor-object relationships . In particular , we annotate higher order relationships , such as ( { waiter } , holding , { pen , sheet } ) , ( { customer , waiter } , on the side of , { dining table } ) . We argue that its outcome , the action hypergraph is more effective when representing multi-object multi-actor interaction . In addition , we provide temporal localization for each sub-level action , and assign a unique ID for each actor/object in a video for temporal tracking . 3.1 Dataset statistics . MOMA contains 373 raw videos at activity level , 2,364 trimmed videos at sub-activity level , and 12,057 atomic action instances . This includes 17 activity classes , 67 sub-activity classes , and 52 atomic action classes . At frame level , we provide action hypergraph annotations for 37 , 428 frames , with 164 , 162 actor/object instances of 20 actor classes and 120 object classes , and 119 , 132 relationship instances of 75 relationship classes . On average , there are 4.39 actors/objects and 3.18 higher-order relationships per frame , 5 instances of atomic actions per clip , and 6.34 instances of sub-activities per untrimmed video ( Table 1 ) . Detailed information is provided in the supplementary material about the distributions of activities , subactivities , atomic activities , entities , relationships , and their co-occurrences . We have adopted a series of protocols to mitigate potential ethical issues of the dataset : ( 1 ) videos are sourced from the YouTube platform , where each video has been cross-verified by multiple annotators to avoid offensive contents ; ( 2 ) we do not provide videos , but instead provide YouTube IDs and download script ; ( 3 ) we control personally identifiable data so that only links to publicly available videos are released , and owners of the video can disable the accessibility at any time ; ( 4 ) any release of imaging data for demonstration purpose will be verified by our group to not contain any personally identifiable information , and will involve using face blurring algorithms . 3.2 Partonomic hierarchy . The MOMA Representation features a four-level partonomic representation for complex activities . Activity . An activity [ 69 ] ( e.g. , dining service ) is a high-level description of an event done by people for a specific purpose . Typically , it is an abstraction of several sub-processes carried out by multiple entities . Spurious features , such as background , often provide contextual clues important for sub-optimal activity classification . Sub-activity . A sub-activity is a stage making up part of a larger activity . Typically , the sub-activities ( e.g. , serve food , clean up the table ) from the same activity may take place within the same environment . Hence , the key to segmenting and classifying a sub-activity is to identify actors , objects , and their interactions . Atomic action . An atomic action [ 22 ] represents an actor ’ s body movement or interaction with other entities . At the most primitive level , atomic actions are actor-centric [ 71 ] , but an actor can simultaneously perform multiple atomic actions . Generally speaking , atomic actions must be spatially and temporally localized in order to be recognized . Action hypergraph . An action hypergraph is a spatial-temporal heterogeneous hypergraph that incorporates static and dynamic information about actors and objects . In an action hypergraph , a node represents an entity ( actor or object ) and a hyperedge ( which can join more than two nodes ) represents a higher-order relationship ( static relationship or dynamic relationship ) between the connected entities . As a spatial-temporal graph , an action hypergraph changes with time [ 50 ] and its nodes are spatially localized and tracked as bounding box tracklets on the raw video . As a result , although the action hypergraph is annotated per frame , based on the granularity of our prediction , it is flexible to be formed into a hypergraph that captures information about higher level actions ( e.g . atomic actions , sub-activities , or activities ) .

Summary: This paper proposes that video understanding be approached by trying to recognize at several different levels of granularity, simultaneously. At the highest temporal granularity is the "activity", which consists of a sequence of temporally-localized "sub-activities". Below that are the "atomic actions" performed by individual actors, and at the bottom an "action hypergraph" that is a scene hypergraph showing the relationships between actors & objects at the frame level.  The authors introduce a video dataset, MOMA, and a task "Activity Parsing" which aims to infer all of these annotations from a video together with ground truth person and object tracks. Further they propose a HyperGraph Action Parsing Network, and compare it to 3DCNNs and graph convolutional networks on the Activity Parsing task.