MOMA: Multi-Object Multi-Actor Activity Parsing
Authors: Zelun Luo
Wanze Xie
Siddharth Kapoor
Yiyun Liang
Michael Cooper
Juan Carlos Niebles
Ehsan Adeli
L. Fei-Fei
Venue: NeurIPS 2021 Poster (2021)
Keywords: activity recognition
video understanding
human-object interaction
temporal action detection

Abstract: Complex activities often involve multiple humans utilizing different objects to complete actions (e.g., in healthcare settings, physicians, nurses, and patients interact with each other and various medical devices). Recognizing activities poses a challenge that requires a detailed understanding of actors' roles, objects' affordances, and their associated relationships. Furthermore, these purposeful activities are composed of multiple achievable steps, including sub-activities and atomic actions, which jointly define a hierarchy of action parts. This paper introduces Activity Parsing as the overarching task of temporal segmentation and classification of activities, sub-activities, atomic actions, along with an instance-level understanding of actors, objects, and their relationships in videos. Involving multiple entities (actors and objects), we argue that traditional pair-wise relationships, often used in scene or action graphs, do not appropriately represent the dynamics between them. Hence, we introduce Action Hypergraph, a spatial-temporal graph containing hyperedges (i.e., edges with higher-order relationships), as a new representation. In addition, we introduce Multi-Object Multi-Actor (MOMA), the first benchmark and dataset dedicated to activity parsing. Lastly, to parse a video, we propose the HyperGraph Activity Parsing (HGAP) network, which outperforms several baselines, including those based on regular graphs and raw video data. 

MOMA: Multi-Objeet Multi-Actor Activity Parsing
Zelun Luo;
Wanze Xie;
Siddharth Kapoor, Yiyun Liang, Michael Cooper,
Juan Carlos Niebles, Ehsan Adeli, Fei-Fei
Stanford University
(alenziuo, wanzexie, siddkap, isaliang, cocpesm), jniebles, eadeli,
https:/ http::/mmmaa.aatttt.oommm
Dining service
Activity
Lidiy
Take the order
Serve food
Clean the table
phantaay
Sub-Activity
(S80
Ulatio accaaa
Rital Graen
Butwer
Buran Pass dar Rlacdayg
Atomic Action
Clamsitey
Wetwo
Castodle plle
Sighad taki Phuisayg
Cliputing soles
Kriall Parsing
Action Hypergraph
Mode:
Bdge:
Figure An overview our paper. We introduce Activity Parsing, a partonomy-ban based activity recognition framework that considers the hierarchy and composition of complex human activities.
Abstract
Complex activities often involve multiple humans utilizing different objects to com- plete actions (e.g., in healtheare settings, physicians, nurses, and patients interact with each other and various medical devices). Recognizing activities poses chal- lenge that requires detailed understanding of actors' roles, objects' affordances, and their associated relationships. Furthermore, these purposeful activities com- prise multiple achievable steps, including sub-activities and atomic actions, which jointly define a hierarchy of action parts. This paper introduces Activity Parsing the overarching task of temporal segmentation and classification of activities, sub-activities, atomic actions, along with an instance-level understanding of actors, objects, and their relationships in videos. Involving multiple entities (actors and objects), we argue that traditional wair relationships, often used in scene action graphs, do not appropriately represent the dynamics between them. Hence, we introduce Action Hypergraph, new representation of spatial-temporal graphs containing hyperedges (i.e., edges with higher-order relationships). introduce Multi- Mulri-Object Multi-Actor (MOMA), the first benchmark and dataset dedicated to activity parsing. Lastly, to parse video, we propose the HyperGraph Activity Parsing (HGAP) network, which outperforms several baselines, including those based on regular graphs and raw video data.
addition,
'These authors contributed equally.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Introduction
While human activity recognition has advanced considerably in recent years, the datasets 8 14 68
and models [69 74, 17
that underlie these improvements still treat human activities as monolithic
events. Typically, the action classification task is only conducted on a coarse temporal scale and on limited set of activity labels. In contrast, real- world activities in domains such as healthcare [87 47
53] surveillance (52 78 29 and entertainment [77 [5 70] complex, which involve intricate social interactions between humans utilizing different objects and sequences of steps they perform to achieve goal. The understanding of these activities requires more fine-grained and structured reasoning beyond single semantic labels. Recently, efforts have been made in developing activity recognition models to address this disparity, including temporal localization [6] (95), hierarchical reasoning [65]. atomic action reasoning [22] 21. and action decomposition [27]. However, none of them fully addresses the needs in real-world applications.
We argue that a complex activity recognition system invokes understanding the detailed hierurchy and composition of activity components in a video sequence [89]. Firstly, human activities are by nature hierarchical [3] and abstract behaviors (such as dining service) are composed of a series of low-level body movements and interactions (such as speaking, sitting, and holding). Defining a vocabulary of activity labels can be ambiguous if multiple levels of granularity (i.c., a hicrarchy) not adequately defined. Secondly, when multiple entities exist in the video, understanding the composition of the video by reasoning each actor's 's social role [6]] each object's 's affordance, and their relationships 27 is crucial to visual dynamics understanding (see Fig. []]
66 for complex human activity recognition. Our proposed activity parsing task parses complex activities into multiple timescales and spatial-temporal grounding of entities and relationships. The representation consists of four levels of hierarchy: (1) activity, (2) sub-activity, (3) atomic action, and (4) action hypergraph. More specifically, an action hypergraph is a spatial-temporal graph [50] tailored for multi-object and multi- actor scenarios, in which hyperedge can encode higher-order relationships by connecting any number of vertices (i.e., entities, including actors and objects). Hyperedges, which can still trivially represent pairwise relationships, define more comprehensive relationships than the pairwise relations prevalent in the traditional scene graphs. In this representation, the contextual information in high-level activities and sub-activities facilitates the fine-grained recognition of scene dynamics the atomic action and action hypergraph levels; low-level action hypergraphs and their constituents provide a grounding for high-level activity recognition necessary for its interpretability and transferability.
Towards this goal, we propose a redefined Action Parsing [39
Furthermore, we propose a novel model, termed Hypergraph Activity Parsing (HGAP) network, for the activity parsing task videos. We also explore other tasks such as video classification and role classification with HGAP. We compare HGAP with an existing graph neural network approach [83] and the sfate-of-the art activity recognition model [16]. By learning with Action Hypergraph, HGAP achieves comparable results with 3D CNN models without pre- trainin on videos from the same domain, and HGAP with X3D backbone can achieve superior results for activity parsing.
Lastly, we introduce the MOMA (Multi-Object,, Multi-Actor) dataset, consisting of 17 activity cat- egories, 67 sub-activity categories, 52 atomic action categories, 120 object categories, 20 actor categories, and 75 relationship (static and dynamic) categories. The MOMA dataset has three unique features: (I) activity annotations with four levels of granularity. which enable activity parsing on different levels of abstraction; (2) multi-object, multi-actor, and categorical labels for actors (i.c., so- cial roles) and objects, which provide exhaustive details of the associated activity; (3) higher-order relationships [12 I between entities, which expressively capture the relationships in crowded and complex scenes. To the best of our knowledge, MOMA is the first dataset with these features.
2 Related work
Action recognition refers to the problem of recognizing human actions in videos, with two major
tasks in the domain (1) action classification [91] and (2) temporal action detection (TAD) 77) For
(1), recent advances are toward 3D CNNs 88 23
6
73
8D such as C3D 73), I3D
g Slow
17 and X3D [16] that jointly model spatial and temporal features. (2) 77 in comparison is a harder
task, and works often follow either a proposal-thens -hasssaaitttnnaaiiaatiionnn paradigm 67
or an
approach to model the two steps at same time 41
88

---

Table 1: A comparison of MOMA with related video datasets. MOMA, the first multi-object, multi-actor activity recognition dataset, offers rich annotations at four levels, with actors and objects represented novel action hypergraph.
Deseed
Action panoromy
viders
hours
i actioon conogeeic
Incalired
Orgeets
chooze
Actoes
cheues
annolated
Kebatinships
edge tyne
chases
Acmityy Ne
14
849
2:40
Kieelcs 700
11
CO0%
1.8K
700
BACS Clips
0.4K
0.8K
200
Social Koles
0.2K
Somerhine
Conertte-SSmmhhingV Somthingg22 21T
120K
174
5723
LEMBAX k0
641
230K
Tok
32"
157
18
85
258
MIONA
375
56
2.4K
Muuic saSion
12K
10
OcTiOn hypergrihh
120 20
teperedge
Today, researchers have pushed human action understanding to deeper level in two major directions that our work is based upon: (1) action as hierarchical representations, (2) action as compositions of visual relationships. To jointly model these two representation, we identify graph neural network and action parsing as two promising tools for our study, briefly surveyed below.
Hierarchical representation learning is often studied on data with network structure 4
90 51
yet its application in video action recognition an emerging direction. Drawing from Cognitive Science studies [89] a hierarchy of human actions can either be taxonomic [4] 11 (e.g.. playing
basketball is a kind of sport), or partonomic 33 35 27 655 (e.g., take an order is a step of dining service). 45 2 leveraged taxonomy hierarchy for action prediction, whereas [35 33
exploited the two-level partonomy of actions for few-shot action classification. This work aims to apply four-level partonomic hierarchy to solve the task of activity parsing. In particular, break high-level activity to sub-activity, then atomic action, and lastly frame-wise representation for visual human-object relationship, as discussed in the next subsection.
Visual relationship and compositionality has been studied video domain to represent human actions. For example, Something-Fiin Else 449] models object relationship tuples, Action Genome formulates human-object relationship in video as a
27
scenegraph, and STAG
views changing relationship as dynamic graph representing an action. To model this, models and new loss functions [93] are proposed. In recent years, graph neural networks [85 (56) have been found to be promising tool for both generating and learning from visual compositions.
25
46
32
38
and has been recently adopted for video understanding. Lines of work have studied its application on spatial- -eetial-eemporl graph repre- sentations such as skeleton data [4 (7) or visual relationships (59 54 More recently, Hypergraph Neural Network [18] was proposed and greatly inspired our work as we see its potential in improving scene graph representations and deepen model's cognitive understanding to human actions.
Graph neural networks (GNNs) were first introduced in 20 64
Action parsing [66] 43 44 39, 558 76 55 775 is often referred to as modeling multiple charac- teristics of actions simultaneously. For example, 39) jointly models human joints, objects, and action labels in images, whereas (44) explored predicting human pose, action attributes and action labels together in videos. In this work, we redefine this task be activity parsing. We will ask the model to predict the activity, to temporally detect the sub-actions that occurred, to localize the people and the objects that involved, and recognize the visual relationships related to the ongoing activity.
3 Data representation and dataset
The MOMA dataset is structured in a four-level hierarchy based on the activity partonomy, with rich annotations at each level. As shown in Figure the activity dining service consists of multiple steps (i.e., sub-activiriis). including take an order, serve wine, serve food, and clean up the table. Further, the sub-activity taking an order also involves multiple entities. consists of several actor-centric atomic actions: sit, talk to someone, write on something, etc. At the lowest level, we provide dense annotation of actor-object relationships. particular, we annotate higher order relationships, such as (Iwaiter), holding, (pen, sheet)), (i (ccustomer,
3

---

waiter), the side of, Idining tablel). We argue that its outcome, the action hypergraph more effective when representing multi-object mullti-ccto interaction. In addition, we provide temporal localization for each sub-level action, and assign unique IDD for each actor/object in video for temporal tracking.
Activity
Semantic relations
140 classes
sa neseers
Sub-activity
Holosym- partonomic (part-of)
75 classes
Hypemym-troe trnnmmm--ypyyymmmmmmm taxoromic (kind-of)
Atomio action
rendann
Actors
Action hypergraph
Static relationships
Nen Me--aenos
Dynamic relationships
Objocts
Entities
Relationships
O
Figure 2: conceptual diagram of the semantic relations introduced in this paper, which includes partonomic and taxonomic hierarchies. The partonomic hierarchy is used for activity parsing, and the taxonomic hierarchy is used for categorizing entities and relationships.
3.1 Dataset statisties
MOMA contains 373 raw videos at activity level, 2,364 trimmed videos at sub-activity level, and 12,057 atomic action instances. This includes 17 activity classes, 67 sub-activity classes, and 52 atomic action classes. At frame level, we provide action hypergraph annotations for 37, 428 frames, with 164, 162 actor/object instances of 20 actor classes and 120 object classes, and 119, 132 relationship instances of 75 relationship classes. On average, there are 4.39 actors/objects and 3.18 higher-order relationships per frame, instances of atomic actions per clip, and 6.34 instances of sub-activites abtiviiiiiees per untrimmed video (Table T]. Detailed information is provided in the supplementary material about the distributions of activities, subactivities, atomic activities, entities, relationships, and their co-oceurrences.
We have adopted series of protocols to mitigate potential ethical issues of the dataset: (1) videos are sourced from the YouTuub platform, where each video has been cross-verified by multiple annotators to avoid offensive contents; (2) we do not provide videos, but instead provide YouTube IDs and download script; (3) we control personally identifiable data so that only links to publicly available videos are released, and owners of the video can disable the accessibility at any time; (4) any release of imaging data for demonstration purpose will be verified by our group to not contain any personally identifiable information, and will involve using face blurring algorithms.
3.2 Partonomic hierarchy
The MOMA Representation features a four-level partonomic representation for complex activities.
Activity. An activity [69 (e.g.. dining service) is a high-level description of an event done by people for specific purpose. Typically, it is abstraction of several sub-processes carried out by multiple entities. Spurious features, such as background, often provide contextual clues important for sub-optimal activity classification.
Sub-activity. A sub-activity is a stage making up part of larger activity. Typically, the sub-activities (e.g., serve food, clean up the table) from the same activity may take place within the same environment. Hence, the key to segmenting and classifying a sub-activity is to identify actors, objects, and their interactions.
Atomic action. An atomic action [2 represents an actor's body movement or interaction with other entities. At the most primitive level, atomic actions are actor-centric (71), but an actor can simultaneously perform multiple atomic actions. Generally speaking, atomic actions must be spatially and temporally localized in order to be recognized.

---

Cantrner
Locking #00.0
Nalker
Bubysiter
Drinng Sill
Sating
Sazng
Tedlng
site al 0.29
Chair
(a) Dining service
(b) Cycling class.
Figure 3: Average action hypergraphs for dining service (Figure 3a and cycling class (Fig- ure 3b). We average all action hypergraphs in an activity class and select the most frequent nodes and edges. A larger hyperedge value indicates a higher frequency within the associated activity.
Action hypergraph. An action hypergraph is spatial-temporal heterogeneous hypergraph that incorporates static and dynamic information about actors and objects. In an action hypergraph, a node represents entity (actor or object) and hyperedge (which can join more than two nodes) represents a higher-order relationship (static relationship or dynamic relationship) between the connected entities. As spatial-temporal graph, an action hypergraph changes with time [50] and its nodes are spatially localized and tracked as bounding box tracklets on the raw video. As a result, although the action hypergraph is annotated per frame, based on the granularity of our prediction, it is flexible to be formed into a hypergraph that captures information about higher level actions (e.g. atomic actions, sub-activities, activities).
3.3 Details of action hypergraph
Action hypergraph is novel spatial-temporal representation at the bottom level of the MOMA hierarchy. features: (1) higher-order relationships that connect multiple entities with hyperedges; (2) class labels for both actors (social roles) and objects (object categories); (3) temporal tracking for entities, resulting in
graphs.
Entity. We define two types of entities, namely actor and object. In an action hypergraph, each entity represented node and defined spatial-temporally (bounding box tracklet) and categorically (actor role object category). Using taxonomic hierarchy, we distinguish three social roles (service provider, service receiver, and independent actor) within 20 actor classes and 13 object categories (kitchenware, clothing. medical supplies, etc.) within 120 object classes. We illustrate the taxonomic hierarchy of entities in Figure and
and provide the complete taxonomy in the supplementary material.
Relationship. We define two types of relationships, namely dynamic relationship and static rela- tionship. A dynamic relationship shows a continued or progressive action initiated by an actor and it usually lasts for a few seconds. On the contrary, a static relationship describes a state of being among two or more entities and its duration is indefinite. In action hypergraph. each relationship represented as an edge and is defined categorically. Unlike previous studies B2 27 13,, we introduce higher-order relationships using hyperedges. The hyperedge provides versatile way to represent single actor's 's body movement (as self-loop), a relationship between two entities (as regular edge, i.e. trivial hyperedge). or a relationship between more than two entities (as non- non-trivial hyperedge). Using a taxonomic hierarchy, we differentiate non-interactive (associated with just one entity) and interactive (associated with more than one entity) movements within 52 dynamic relationship classes. Similarly, we distinguish spatial, physical, and aftentional relationships within 23 static relationship classes. We illustrate the taxonomic hierarchy of relationships in Figure 2 and provide the complete taxonomy in the supplementary material.
Significance of higher-order relationships. Mathematically, regular graph can be converted to a hypergraph by merging edges that have same attributes and come from same nodes. However, this is not the case for MOMA dataset. human interactions, higher-order relationships contain strictly more information than its pair-wise equivalents. For example, during annotation, (waiter, looking at, glass), (waiter, pouring water into, glass), is likely to separated from (two customers, looking at, glass). Here, different instances of looking at are not merged be- cause they serve different purposes and do not share same intention from the actors. More examples shown in the supplementary section.
5

---

Tracokett
Hypengraph
Graph Representation
Convolution Medale
Spatial
Actor-centric
Reedoot
Pole CLS
Graph Stream
Tamparal
Frame-
Reedintt
Object Traskinng
Relationship Atomic Action CLS
Mldulle
Video Stream
Video Reprosentation
Spatial-
Video-level
Activity, Sub- activityy
Node Video Pooling
Multi- label Atomic Action CLS
Video Encoding Module
Raw Videos
The structure of the HGAP model. In activity parsing tasks, we design the multi-head MLP with different type of output heads for different prediction result, which is covered in detail in Section 4.2. CLS stands for classification.
Figure 4:
Method
We introduce the HyperGraph Action Parsing (HGAP) network, which uses the action hypergraph representation foundation for activity parsing, hicrarchical action classification, temporal seg- mentation, and actor role classification (Figure
4.1 Action hypergraph representation
The action hypergraph of video of length is defined G (V,E, T (t()) ((),, ........ the set of node types. In videos, a node type corresponds to an actor role or an object category. is the set of edge rypes. An edge type corresponds to a relationship category. V = {ui, ... is set of nodes. A node correspond to specific actor object instance. = [E,,E,.... E.....I . is length-n list of sets, one for each frame. Each set E; (es,1,6,,2, ... is set of directed hyperedges, each of which e,, = [.j .Ty, Vilg defined by tuple of head nodes Vhead= C V, tail nodes i. C V. and relationship type ry E R, where E [1, n] and E [1, E&a] directed hyperedge corresponds to a high-order relationships between two sets of entities in specific frame.
R = =f(1), ,((2,
4.2
An action hypergraph is graph that (1) hypergraph (i.e., hyperedge can join any number of nodes); (2) heterogeneous (i.e., contains different types of nodes); (3) dynamic (i.e., the graph changes with time). More specifically, in action hypergraph, only the set of hyperedges evolve over time and the set of nodes remains constant. An actor or object that has disappeared or has not yet appeared in the scene have the degree
4.2 HGAP network
The HGAP Network is two stream network that combines a graph branch for modeling action hypergraph, and video branch that serves as backbone to extract the RGB features across the video. In this section, we describe our implementation based for the graph branch.
Given a trimmed video = {i(), ((2) (N)) of frames, the model outputs set of n, bounding box tracks of actors T. {to))
(1) 1/2 .... (ma)), and n. bounding box tracks of object instances consists
T, = (())) (2) ...,,
sequence of bounding boxes. Note that assume the availability of actor and object bounding boxes (but not the categories)
(6 (m.)) }. Each track t = {6(2), ((2), ((2),.... (....,()))
.,2),...,(")) (m))
because object detection 63 62) has achieved impressive result and is beyond the scope of this paper.
Hypergraph modeling. To represent the hypergraphs, we propose to utilize hypergraph convolution borrowed from hypergraph neural networks (HGNN) [18]. Note that HGAP's graph branch can be replaced with any grahp-based feature extraction model. Hence, baseline model, we use Graph Convolutional Networks (GCN) 31 to compare our results with, Below we describe the setup of our general graph encoder and its readout functions.

---

Nodes. Nodes are defined as the entities (actors and objects) in trimmed video. We further augment the node features with image features extracted from the bounding box regions, which can also be replaced with region proposal features from models (Faster R-CNN 663) for end-to-end training.
Edges. Edges are defined the relationships between entities in a trimmed video. In one of tasks, the goal is to predict the category of edges in the graph.
Readouts. Readouts are defined with three strategies for different output heads. We first use MLP to map node features to hidden dimension with the same encoder, and then we have actor-centric readout, frame-level readout, video-level readout, each designed for specifie tasks.
(1) Actor-centric readout applies temporal average pooling for actor node's 's features across the entire video. This readout function is followed by role prediction heads for each actor across the video. (2) Frame-level readoul applies global average pooling across for all nodes in the graph. This readout connects the per-frame classification heads for relationship classification and atomic action localization.
(3) Video-level readout first performs nodes pooling to generate embedding for each graph, and then applies temporal average pooling to gather the information of all graphs in the video. This readout designed for all video-level tasks including activity classification, sub-activity classification, and multi-label atomic action classification.
Hyperedge convolution layer. In HGAP Network, we follow the original design [18 of hyperedge convolution operator and build the convolution layer fXX, W, θ) as follows
x(+1)) = (D. HWD, H" D. x()
in which X E R refers to an embedding matrix with N nodes of feature channels, De. D. defines the vertex and edge degree matrices of the hypergraph, and θ is the weight matrix in the update function. In our model, we use the bounding box info and the average pooled region of interest (Rol) as input feature for the actor and object nodes, with an equal weight W for all hyperedges, and ReIU activation for a.
We show that by leveraging the tracking information and creating a we are able to capture more nuance in the actions with short duration and yield better outcome for hierarchical predictions.
hypergraph,
4.3
Video feature extraction
The video branch of the HGAP network is built upon the state-of-the-ar action recognition model X3D 16 shows promising performance on a variety of video action datasets including Charades (68),
AVA 22 and Kinectics [30]. We use the original X3D_ I. baseline approach for hierarchical action classification, and compare it with HGAP's performance on same tasks.
The input to the video model includes a trimmed video sequence = {((()) (())),()......... ((M) (()....()) of N image frames with sample rate of 8 with N = 16. We pretrain the X3D L model on Kinetics- 400 [30], and then modify the original X3D_L. model with a multi-head output to enable 3-level hierarchical action classification for activity, sub-activity, and atomic actions. In particular, we perform multi-class single-label classification for activity and sub-activity level output, and perform multi-class multi-label classification for a set of atomic actions that show up in the entire trimmed video. We train this video branch independently, and pick the model with best validation result across the three tasks.
To incorporate the video feature extraction backbone to the HGAP network, we concatenate the output heads of X3D with the video-level output heads of the graph branch, which include the activity classification head, sub-activity classification head, and multi-label atomic action classification head. At test time, we average the logits of the two branches together and generate the prediction output.
5 Experiments
In this section, we examine three different tasks using the MOMA dataset. The first task is activity parsing, which fully leverages the strength of our dataset. Second, we conduct experiments on the hierarchical classification task to compare HGAP with the state-of-the-art action classification model. Finally, consider role classification, which is rarely discussed in existing work but critical to
7

---

Table 2: The performance of HGAP Network activity parsing. "Oracle" indicates that ground truth graph structure (without class label for node and edge) as input to the network. For GCN not capable of handling hyperedges, we show its result for pair- wase edge classification.
(All results reporteed as MAP %
Level
Activity Clasolication
Sub-activity Clavsification
Atouic Action
Action Hypergraph Role Classilicaiion
Methods
Claseliaaionn
Localization
Hyperolge Classiliction
Edge Ciaselication
GUN
X0
374
29.4
29.1
64.7
37.7
HGAP
73.9
42.5
29.2
50.3
27.6
46.4
HGAP Oraele
88.4
31.3
32.7
54.3
understanding video semantics. For all three experiments, we split 80% of data for training, and hold out the rest for validation. The major hyperparameter we focus on is the loss weight assigned for each output head, while we also tune on model hidden size and initial learning rate. We use an 8-GPU (Tesla V100) environment for training video feature extractor and fine-tuning on the action hypergraph. We demonstrate the significance of action hypergraph representation through these three experiments and the carefully designed baselines.
5.1 Activity parsing
The purpose of the activity parsing task is to demonstrate that the model has comprehensive cognitive understanding of an activity. Specifically, we train a model capable of detecting and classifying activities, sub-activities, atomic actions, well as action hypergraphs within the video. In addition to raw video data, we assume bounding box tracklets of people and objects in the scene available. This paper will not concentrate object detection and tracking since other studies 663 186)6 92] have demonstrated excellent performance. Instead, the paper focuses specifically on temporal reasoning. We also perform Oracle experiment with ground-truth structures of action hypergraphs (without knowing the classes of nodes and edges). The results show that an accurate action hypergraph crucial for the overall activity parsing performance.
Problem formulation. Activity parsing is defined as a joint prediction of the following characteristics: (1) activity category; (2) sub-activit category; (3) atomic action categories with temporal localization; (4) an action hypergraph comprised of all nodes' and edges' categories. We define inputs to be (i) a trimmed video with a single sub-activity that is segmented from the complete video, and (ii) tracklets that represent actors and objects across the trim video. Tracklets are defined as temporally stacked Rol for the actor or object in the video.
Evaluation metries. We find that mean average precision (mAP) (26 is a good metric for all sub-tasks defined above. It is suitable for multi-class classification when sampling is spread out over a long tail distribution like in our case. For the classification of atomic actions and actor roles, multi-label mAP (68) is used. We also adopted it for the task of localizing temporal atomic actions, because atomic actions are highly overlapped both spatially and temporally, as multiple people can perform different atomic actions the same time. Thus, each frame is predicted with set of atomic actions, and mAP is computed for each frame at granularity of one frame per second.
Since activity parsing is a newly proposed task, there are no existing published baselines. However, our objective is demonstrate the utility of the action hypergraph representation, and our baseline is therefore modified from the Graph Convolutional Network (GCN) model 31 to take in Rol as node feature, and train with multiple output heads, where hyperedges are decomposed to pair-wise ones.
Baselines.
Results. Table reports activity parsing experiment performance. HGAP achieves better or com- parable performance compared to GCN on all high-level tasks including classification for activity, sub-activity and atomic actions, along with atomic action temporal localization. Interestingly, HGAP performs better on all high level tasks even though GCN performs better on graph level classifications. We have seen evidence in the image and video domain that the learning of visual relationships increases performance on high-level tasks such as image captioning (32), collection detection [25], and action recognition 13. 277. This thus illustrates that hypergraphs are intrinsically harder to learn, but their dense feature representation enables the model to better understand the video semantics, even the hypergraph itself is not sufficiently learned. HGAP Ora cl e shows that greater performance gains with better hypergraph model.
8

---

Table 3: The performance of HGAP network on the hierar-
Table The performance of HGAP
chical activity classification task.
network on role classification in com-
(All results are reported as mAP %)
parison with ResNet-50.
Method
Pre-train
Activity
Sub-activity
Atomic Action
Method
Pre-train
mAP %
X3D [16
Kinetics-000 [30
70.9
40.0
27.1
X3D + GCN
Kinetics-400
71.8
41,9
29.7
ResNet-50 24
ImageNet [10
37.9
HGAP
Kinetics-400
73.9
42.5
29.2
HGAP
Kinetics-400
46.4
HGAP Oracle
Kinetics-400
88.4
44.1
31.3
HGAP Oracle
Kinetics-400
54.3
5.2
Hierarchical action classification
In this experiment, we further evaluate the utility of the action hypergraph representation. Since the proposed HGAP combines video information with learned action hypergraphs, we hope to show that HGAP achieves performance improvement on the action classification task, even when compared with the state-of-the-art action model augmented with a regular graph.
Problem formulation. To fully assess the model's 's capability to understand ongoing activity, we evaluate a joint multiclass classification across three action levels. We use the same inputs as in activity parsing, but we predict (1) activity categories (single-label, multi-class), (2) sub-activity categories (single-label, multi-class), and (3) atomic action categories (multi-label, multi-class). mAP is used as the metric for sub-tasks (1) and (2), and multi-label mAP for sub-task (3).
Baselines. We implemented baselines. The first baseline is X3D_L model [T6] that shows state- of-the-art performance on several challenging action datasets, but customized with multi-headed output layer that incorporates RGB video along with the flattened graph matrix. The second is a
X3D L model enhanced with GCN, which allows it to learn structured graph features from the action hypergraph (with hyperedges broken to pair-wise ones).
Results. As shown in Table 3 HGAP outperforms X3D on all three sub-tasks, and achieves a better average performance than X3D combined with GCN, with 2.1% mAP margin the activity classification sub-task. While HGAP Oracle performs even better, it is important to note that this an upper bound, not only because did use ground-truth actor or object labels, but also because ground-truth graph structure is sampled at a rate of second per frame.
5.3 Role classification
In video datasets, role classification is rarely defined except briefly touched upon in 61 28 Using dense annotations and comprehensive actor role categories in MOMA, we aim highlight this problem as a stand-alone task and promote research direction to the joint modeling of actor-centric actions and actor roles within ongoing activity.
Problem formulation. In this experiment, we investigate how understanding visual relationships in the on-going activity enhances the model's ability to recognize each actor's role in the video. The inputs are identical to those in activity parsing, but the output is a multi-label, multi-class role prediction for all the people in the video. Our prediction is based on the whole actor tracklet instead of single frame, by first aggregating all Rols with the temporal tracking ID, and then classifying the actor's role. mAP is used as evaluation metric for this task.
classifier based on ResNet-50 which accepts tracklets input. In this algorithm, tracklets are processed distributions for each frame in each tracklet, and then the average of these distributions across frames is used to predict actor roles for each tracklet. To emphasize the importance of visual relationships, this baseline does not use graph structure on purpose.
Baseline.
The baseline for this evaluation
frame-level produce role class
Results. Table 4 compares the baseline model to two proposed models, with HCAP outperforming
ResNet-50 by 8.5%. Further, HGAP Oracle can improve mAP scores by 7.9% when the ground-truth graph structure is applied. The results indicate that learning good action hypergraph representation indeed useful for role classification, and we can expect that a more advanced hypergraph modeling technique will result in even greater improvements.

---

6 Conclusion
We created first-of-its-knnd video dataset, MOMA, that (1) identifies list of activities involving multi-person collaboration, (2) formulates four-level hierarchical representation of activity and (3) proposes Action Hypergraph, spatial-temporal heterogeneous hypergraph representation for visual relationships. Furthermore, We defined activity parsing, new task to show model's holistic understanding of an activity, and designed HGAP, new urchitecture that combines video feature extraction with hypergraph neural networks and achieves superior result on variety of tasks. We anticipate that MOMA opens up a wide range of potential future research directions, and helps future works toward better action recognition and with providing the means for holistic and detailed understanding of the videos. Detailed and structured understanding of activities and their components (i.e., Activity Parsing) as offered by MOMA, when combined with large-scale unlabeled data, may facilitate few-shot or even zero-shot action recognition.
Acknowledgment
This work was partially supported by the Schmidt Futures, Toyota Research Institute (TRI), and National Science Foundation grant 2026498. This study also benefited from Stanford Institute for Human-Centered AI (HAT) AWS Cloud Credits.
References
[1] Sameer Agarwal. Kristin Branson, and Serge Belongic. Higher order learning with graphs. In Proceedings of the 23rd international conference on Machine learning, pages 17-24, 2006.
[2] Humam Alwassel, Fabian Caba Heilbron, and Bernard Ghanem. Action search: Spotting actions in videos and its application to temporal action localization. In Proceedings of the European Conference Computer Vision (ECCV), pages 251-266, 2018.
[3] Roger G Barker and Herbert F Wright. Midwest and its children: The psychological ecology of an american town. 1955.
[4] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 1798) 1828, 2013.
[5] Shyamal Buch, Victor Escorcia, Bernard Ghanem, Li Fei-Fei, and Juan Carlos Niebles. End-to- end, single-stream temporal action detection in untrimmed videos. Proceedings of the British Machine Vision Conference (BMVC), 2017.
[6] Shyamal Buch, Victor Escorcia, Chuanqi Shen, Bernard Ghanem, and Juan Carlos Niebles. Sst: Single-stream temporal action proposals. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 291 2911-2920, 2017.
[7] Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics- 700 human action dataset. arXiv preprint /907.v09900006900: 2019.
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 0299-6308, 2017.
[9] Haochen Chen, Bryan Perozzi, Yifan Hu, and Steven Skiena. Harp: Hierarchical representation learning for networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
[10] Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages
2009.
[11] Ali Diba, Mohsen Fayyaz, Vivek Sharma, Manohar Paluri, Jürgen Gall, Rainer Stiefelhagen, and Lue Van Gool. Large scale holistic video understanding. arXiv preprint arXiv: 1904. / 1451, 2019.
10

---

[12] Júlia Domingo, Guillaume Diss, and Ben Lehner. Pairwise and higher-order genetic interactions during the evolution of trna. Nature, 558(7708):117-121, 2018.
[13] N. Rai et al. Home action genome: A large-scale multi-view video dataset of daily activities at home. In https://homeectttneennne.oog 2021.
[14] Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. Activitynet: large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 961-970, 2015.
[15] H. Fan, Li Meng-jun, and Li Jin-jiang. Action parsing based on bag-of bag-of-words: 2015.
[16] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203-213, 2020.
[17] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 0202-6211, 2019.
[18] Yifan Feng, Haoxuan You, Zizhao Zhang, Rongrong Ji, and Yue Gao. Hypergraph neural networks. Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3558-3565, 2019.
[19] Jiyang Gao, Zhenheng Yang, Kan Chen, Chen Sun, and Ram Nevatia. Turn tap: Temporal unit regression network for temporal action proposals. In Proceedings of the IEEE international conference on computer vision, pages 3628-3633 2017.
[20] M. Gori, G. Monfardini, and F. Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference Neural Networks, 2005., volume pages 729-734 vol. 2, 2005.
[21] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision, pages 5842 5850, 2017.
[22] Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sud- heendra Vijaanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et Ava: video dataset of spatio-temporally localized atomic visual actions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6047-6056, 2018
[23] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history 2d cnns and imagenet? In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6546-6555, 2018.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pultern recognition, pages 770-778, 2016.
[25] Roei Herzig, Elad Levi, Huijuan Xu, Hang Gao, Eli Brosh, Xiaolong Wang, Amir Globerson, and Trevor Darrell. Spatio-temporal action graph networks. In Proceedings of the IEEE/CVE International Conference Computer Vision Workshops, pages 0-0, 2019.
[26) Haroon Idrees, Amir Zamir, Yu-Gang Jiang, Alex Gorban, Ivan Laptev, Rahul Sukthankar, and Mubarak Shah. The thumos challenge on action recognition for videos "in the wild". Computer Vision and Image Understanding, 155:1 1-::1223, 2017.
[27] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 10236-10247, 2020.
11

---

[28] Baoxiong Jia, Yixin Chen, Siyuan Huang, Yixin Zhu, and Song- Cong-Chun Zhu. Lemma: A multi- view dataset for learning multi-agent multi-task activities. In European Conference on Computer Vision, pages 767-786. Springer, 2020.
[29] Cheng-Bin Jin, Shengzhe Li, and Hakil Kim. Real-time action detection in video surveillance using sub-action descriptor with multi-cnn. arXiv preprint 171000337 2017.
[30] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 06950, 2017.
[31] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02900) 2016.
[32] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1))3 2017.
[33] Tian Lan, Tsung-Chuan Chen, and Silvio Savarese. A hierarchical representation for future action prediction. In European Conference on Computer Vision, pages 689-704. Springer, 2014.
[34] Tian Lan, Yuke Zhu, Amir Roshan Zamir, and Silvio Savarese. Action recognition by hier- archical mid-level action elements. In Proceedings of the IEEE international conference on computer vision, pages 4552 4560,, 2015.
[35] Oscar D Lara and Miguel A Labrador. A survey on human activity recognition using wearable sensors. IEEE communications surveys & tutorials, 15(3):1192- 2012.
[36] Ang Li, Meghana Thotakuri, David A Ross, João Carreira, Alexander Vostrikov, and An- drew Zisserman. The ava-kinetics localized human actions video dataset. arXiv preprint arXiv:2005.002// 2020.
[37] Maosen Li, Siheng Chen, Xu Chen, Ya Zhang, Yanfeng Wang, and Qi Tian. Actional-structural graph convolutional networks for skeleton-based action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3595-3603, 2019.
[38] Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang. Scene graph generation from objects, phrases and region captions. In Proceedings of the IEEE International Conference on Computer Vision, pages 1261-1270, 2017.
[39] Zhujin Liang, Xiaolong Wang, Rui Huang, and Liang Lin. An expressive deep model for human action parsing from single image. In 2014 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6. IEEE, 2014.
[40] Tianwei Lin, Xiao Liu, Xin Li, Errui Ding, and Shilei Wen. Bmn: Boundary-matching network for temporal action proposal generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3889-3898, 2019.
[41] Tianwei Lin, Xu Zhao, and Zheng Shou. Single shot temporal action detection. In Proceedings of the 25th ACM international conference on Multimedia, pages 988-996 999999967779 2017.
[42] Tianwei Lin, Xu Zhao, Haisheng Su, Chongjing Wang, and Ming Yang. Bsn: Boundary sensitive network for temporal action proposal generation. In Proceedings of the Europeun Conference on Computer Vision (ECCV), pages 3-19, 2018.
[43] L. Liu, Yi Zhou, and L. Shao. Deep action parsing in videos with large-scale synthesized data. IEEE Transactions on Image Processing, 27:2869-2833 2018
[44] Li Liu, Yi Zhou, and Ling Shao. Dap3d-net: Where, what and how actions occur in videos? In 2017 IEFE International Conference on Robotics and Automation (ICRA), pages 138-145 IEEE, 2017.
12

---

[45] Teng Long, Pascal Mettes, Heng Tao Shen, and Cees GM Snoek. Searching for actions on the hyperbole. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1141-1130, 2020.
[46] Cewu Lu, Ranjay Krishna, Michael Bernstein, and Li Fei-Fei. Visual relationship detection with language priors. European conference on computer vision, pages 852-869. Springer, 2016.
[47] Zelun Luo, Jun-Ting Hsieh, Niranjan Balachandar, Serena Yeung, Guido Pusiol, Jay Luxenberg. Grace Li, Li-Jia Li, N Lance Downing, Arnold Milstein, et al. Computer vision-based descriptive analytics of seniors' daily activities for long-term health monitoring. Machine Learning for Healthcare (MLHC), 2, 2018.
[48] Srikanth Malla, Behzad Dariush, and Chiho Choi. Titan: Future forecast using action priors. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11186-1119, 2020.
[49] Joanna Materzynska, Tete Xiao, Roci Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something Compositional action recognition with spatial-tempone interaction networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1049-1059, 2020.
[50] Othon Michail. An introduction to temporal graphs: An algorithmic perspective. Internet Mathematics, 12(4):239-280, 2016.
[51] Maximilian Nickel and Douwe Kiela. Poincar 'e embeddings for learning hierarchical repre- sentations. arXiv preprint arXiv:1705.08033 2017.
[52] Sangmin Oh, Anthony Hoogs, Amitha Perera, Naresh Cuntoor, Chia-Chih Chen, Jong Taek Lee, Saurajit Mukherjee, JK Aggarwal, Hyungtae Lee, Larry Davis, et al. large-scale benchmark dataset for event recognition in surveillance video. In CVPR 2011, pages 3153-3160 IEEE, 2011.
[53] Venet Osmani, Sasitharan Balasubramaniam, and Dmitri Botvich. Human activity recognition in pervasive health-care: Supporting efficient remote collaboration. Journal of network and computer applications, 3144):228-65, 2008.
[54] Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles. Spatio-temporal graph for video captioning with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10870-10879, 2020.
[55] Mingtao Pei, Yunde Jia, and Song-Chun Chun Zhu. Parsing video events with goal inference and intent prediction. In 2011 International Conference on Computer Vision, pages 487-494, 2011.
[56] Mengshi Qi, Weijian Li, Zhengyuan Yang, Yunhong Wang, and Jiebo Luo. Attentive relational networks for mapping images to scene graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3957-3966, 2019.
[57] Mengshi Qi, Yunhong Wang, Jie Qin, Annan Li, Jiebo Luo, and Lue Van Gool. stagnet: an attentive semantic rnn for group activity and individual action recognition. IEEE Transactions on Circuits and Systems for Video Technology, 30(2):549- 5655 2019.
[58] S. Qi, B. Jia, S. Huang, P. Wei, and S. C. Zhu. A generalized earley parser for human activity parsing and prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 1-1, 2020.
[59] Xufeng Qian, Yueting Zhuang, Yimeng Li, Shaoning Xiao, Shiliang Pu, and Jun Xiao. Video relation detection with spatio-temporal graph. Proceedings of the 27th ACM International Conference on Multimedia, pages 84 94-93, 2019.
[60] Zhaofan Qiu, Ting Yao, and Tao Mei. Learning spatio-temporal representation with pseudo-3d residual networks. In proceedings of the IEEE International Conference on Computer Vision pages 5533-5541, 2017.
13

---

[61] Vignesh Ramanathan, Bangpeng Yao, and Li Fei-Fei. Social role discovery in human events. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 2475-2482 2013.
[62] Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEFE conference computer vision and pattern recognition, pages 779-788, 2016.
[63] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv: 2015.
[64] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008
[65] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Finegym: A hierarchical video dataset for fine-grained action understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3616-2625, 2020.
[66] Dian Shao, Yue Zhao, Bo Dai, and Dahua Lin. Intra- and inter-action understanding via temporal action parsing. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[67] Zheng Shou, Dongang Wang, and Shih-Fu Chang. Temporal action localization in untrimmed videos via multi-stage cnns. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1049-1055 2016.
[68] Gunnar A Sigurdsson, Güi Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in homes: Crowdsoureing data collection for activity understanding. In European Conference on Computer Vision, pages 510-526. Springer, 2016.
[69] Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. arXiv preprint arXiv: 2014.
[70] Khurram Soomro and Amir R Zamir. Action recognition in realistic sports videos. In Computer vision in sports, pages 181-208. Springer, 2014.
[71] Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric relation network. In Proceedings of the European Conference on Computer Vision (ECCV), pages 318-334, 2018.
[72] Dídac Surís, Ruoshi Liu, and Carl Vondrick. Learning the predictability of the future. arXiv preprint arXiv:2101. 01600, 2021.
[73] Graham W. Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler. Convolutional learning of spatio-temporal features. In Kostas Daniilidis, Petros Maragos, and Nikos Paragios, editors, Computer Vision ECCV 2010, pages 140-153, Berlin, Heidelberg, 2010. Springer Berlin Heidelberg.
[74] Limin Wang, Yuanjun Xiong. Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang. and Lue Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In European conference on computer vision, pages 20-36. Springer, 2016.
[75] Ping Wei, Yang Liu, Tianmin Shu, Nanning Zheng, and Song- Chun Zhu. Where and why are they looking? jointly inferring human attention and intentions in complex tasks. In 2018 IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 6801-6809, 2018.
[76] Ping Wei, Yibiao Zhao, Nanning Zheng, and Song- Zhu. Modeling 4d human-object interactions for event and object recognition. 2013 IEEE International Conference on Computer Vision, pages 3272-3279, 2013.
[77] Daniel Weinland and Edmond Boyer. Action recognition using exemplar-based embedding. In 2008 IEEE Conference Computer Vision and Pattern Recognition, pages 1-7, IEEE, 2008.
14

---

[78] Zhenyu Wu, Zhangyang Wang, Zhaowen Wang, and Hailin Jin. Towards privacy- privacy-preserviee preserving visual recognition via adversarial training: A pilot study. In Proceedings of the European Conference Computer Vision (ECCV), pages 606-624, 2018.
[79] H. Xia and Y. Zhan. A survey on temporal action localization. IEEE Access, 8:70477-70487, 2020.
[80] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy, Rethinking spa- tiotemporal feature learning: Speed-accuracy trade-offs in video classification. In Proceedings of the European Conference on Computer Vision (ECCV), pages 305-321, 2018.
[81] Yuanjun Xiong, Yue Zhao, Limin Wang, Dahua Lin, and Xiaoou Tang. pursuit of temporal accuracy in general activity detection. arXiv preprint 1003,0/7000027166 2017.
[82] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5410-5419 2017.
[83] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural 2018.
networks? arXiv preprint
[84] Sijie Yan, Yuanjun Xiong, and Dahua Lin. Spatial temporal graph convolutional networks for skeleton-based action recognition. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[85] Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. Graph r-cnn for scene graph generation. In Proceedings of the European conference computer vision (ECCV), pages 670-685, 2018.
[86] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European conference on computer vision (ECCV), pages 684 699, 2018.
[87] Serena Yeung, Francesca Rinaldo, Jeffrey Jopling, Bingbin Liu, Rishab Mehra, N Lance Downing, Michelle Guo, Gabriel M Bianconi, Alexandre Alahi, Julia Lee, et al. A computer vision system for deep learnin-bbased detection of patient mobilization activities in the icu. NPJ digital medicine, 2(1):1-5, 2019.
[88] Serena Yeung, Olga Russakovsky, Greg Mori, and Li Fei-Fei. End-to-end learning of action detection from frame glimpses in videos. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2678-2683 2687, 2016.
[89] Jeffrey M Zacks and Barbara Tversky. Event structure in perception and conception. Psycho- logical bulletin, 127(1):3, 2001.
[90] Daokun Zhang, Jie Yin, Xingquan Zhu, and Chengqi Zhang. Network representation learning:
A survey. IEEE transactions on Big Data 6(1):3228, 2018.
[91] Hong-Bo Bo Zhang, Yi-Xiang Zhang, Bineng Zhong, Qing Lei, Lijie Yang. Ji-Xiang Du, and Duan-Sheng Chen. A comprehensive survey of vision-based human action recognition methods. Sensors, 1955, 2019.
[92] Zhang, Mohamed Elhoseiny, Scott Cohen, Walter Chang, and Ahmed Elgammal. Relationship proposal networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5224,5234, 2017.
[93] Ji Zhang, Kevin Shih, Ahmed Elgammal, Andrew Tao, and Bryan Catanzaro. Graphical contrastive losses for scene graph parsing. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 1 11535-1 1143,, 2019.
[94] Hang Zhao, Antonio Torralba, Lorenzo Torresani, and Zhicheng Yan. Hacs: Human action clips and segments dataset for recognition and temporal localization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8668-8678, 2019.
15

---

[95] Yue Zhao, Yuanjun Xiong, Limin Wang, Zhirong Wu, Xiaoou Tang, and Dahua Lin. Temporal action detection with structured segment networks. In Proceedings of the IEEE International Conference Computer Vision, pages 2914-2923, 2017.
7 NeurIPS 2021 Paper Checklist
The NeurlPS Paper Checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. For each question in the checklist:
You should answer yes, no, or n/a.
You should reference the section(s) of the paper that provide support for your answer.
You can also optionally write a short (1-2 sentence) justification of your answer.
7.1 For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Yes. Paper's contribution is summarized in abstract, and reflected in further detail in the last paragraph of the introduction section.
(b) Have you read the ethics review guidelines and ensured that your paper conforms to them?
Yes, we can confirm.
(c) Did you discuss any potential negative societal impacts of your work?
Yes, we have touched upon the society impact of our work in both ways (positive application: healthcare, negative application: surveillance), as highlighted in the first paragraph of the introduction. Regulations and transparency of work's 's application are deemed as necessary to mitigate such impact.
(d) Did you describe the limitations of your work?
Yes, as highlighted in result in section 5 of the paper. The proposed model has limited capability in predicting hypergraph, yet show how such limitation in turn demonstrates the effectiveness of the hypergraph representation for supporting high-level tasks.
7.2
If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results?
n/a
(b) Did you include complete proofs of all theoretical results?
n/a
7.3 If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either the supplemental material or as a URL)?
Yes. Steps to reproduce experimental results covered in the supplemental material. Code, data, and further instructions will be released at //moma. www.ford.dd.cooo edw.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
Yes. They have been summarized in first paragraph of section 5, and explained in detail in the Problem Formulation and Baseline section.
(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
16

---

No, we don't t report error bars, confidence intervals, or statistical significance tests as they are irrelevant supporting our main claim.
(d) Did you include the amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)?
Yes, we have specified them first paragraph section 5.
7.4 If you are using existing assets (e.g,, code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators?
n/a. The proposed asset is one the major contributions our paper.
(b) Did you mention the license of the assets?
n/a.
(c) Did you include any new assets either in the supplemental material or as a URL?
Yes, the proposed MOMA dataset will be released at piltps://monnee sta.imadaaaafodd......m..
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating?
n/a. We do not collect data from any particular individual, and we only provide access to the publicly available data, whose owner can choose to disable accessibility any time.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
Yes, as it has been covered in detail in section 3.1 of the paper.
7.5 If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if appli- cable?
n/a
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
n/a
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
n/a
17


Reviews:
[{"review_id": "V5LRGWCKLnp", "reviewer": "Reviewer_PT1W", "summary": "This paper proposes that video understanding be approached by trying to recognize at several different levels of granularity, simultaneously. At the highest temporal granularity is the \"activity\", which consists of a sequence of temporally-localized \"sub-activities\". Below that are the \"atomic actions\" performed by individual actors, and at the bottom an \"action hypergraph\" that is a scene hypergraph showing the relationships between actors & objects at the frame level. \nThe authors introduce a video dataset, MOMA, and a task \"Activity Parsing\" which aims to infer all of these annotations from a video together with ground truth person and object tracks. Further they propose a HyperGraph Action Parsing Network, and compare it to 3DCNNs and graph convolutional networks on the Activity Parsing task.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "[Originality] Although there have been a number of recent proposals formulating \"temporal scene graphs for video\", this paper provides a new perspective that I think is valuable, in particular because it unifies several video tasks -- classification, action localization, trimmed clip classification, spatiotemporal atomic actions, and visual relationships -- all under a single umbrella and embodied in a single dataset.\n\n[Quality] The paper is generally very good. In particular the dataset represents a significant investment and seems valuable to the community. The HGAP model seems not particularly compelling due to uneven performance on the task, but that's fine for a dataset baseline. However my main question is around the ability to infer hypergraphs from video. For the technique to be useful, ultimately they will need to be inferred from raw, unannotated video, but in the paper's experiments, the graphs are given as input. Realizing the many degrees of freedom a hypergraph has, how reasonable is it that we can expect a model to be able to infer them from raw data? If inferring reasonable hypergraphs is very hard, then the modeling approach falls down. It would be nice to see how sensitive the approach is to errors in inferred hypergraphs; initial results in Table 2 (Oracle vs. non-Oracle) suggest that it might be very sensitive.\n\nSimilarly, it would be nice to see how reliably human annotators can produce a consistent, agreed-upon hypergraph for a video. This would be a good upper-bound for how well we think a machine could do it -- if people can't agree on the correct hypergraph, how could a machine do it?\n\n[Clarity] The paper is clearly written, and generally well organized. However it's a little confusing that the experimental section is divided into three tasks. Upon digging deeper it's apparent that two of the tasks--5.2 Hierarchical Action Classification and 5.3 Role classification--are simply subtasks of 5.1 Activity Parsing, so it could just be a single task.\n\nAnother point that could be more clear is that the actual hypergraph structures are provided as input to HGAP / GCN, and not inferred directly from the input videos. I realized this only after seeing \"action hypergraph generation from video\" listed as future work in the appendix.\n\n[Significance] I think the dataset is particularly valuable since connects coarse activities, to atomic actions, and to video scene graphs, which could enable research into more \"holisitic\" video understanding methods. Although its size is currently moderate, the authors note that it will soon contain 5x more annotated data, which will be impressive. Judged mainly on the significance of the dataset, I think this paper is valuable.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not addressed. Any video action understanding model can be applied to the task of surveillance, although the labels introduced in this dataset seem benign.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {"ethical_concerns": "This paper doesn't investigate or address potential biases in the dataset (e.g. gender, race, age), which I think could crop up if anyone tried to apply a model trained on this benchmark to a real-world scenario. The authors could add a disclaimer accordingly."}}, {"review_id": "CiLRHk2hssi", "reviewer": "Reviewer_hk9t", "summary": "The paper introduces a new dataset for video-based activity parsing. Instead of assigning a single action label to a video, the paper decomposes an activity into sub-activities and atomic actions. Instance-level annotations of actors, objects and their relationships are also annotated. Based on this dataset, the paper proposes a new model, action hypergraph, for activity parsing. Experimental results demonstrate the potential applications of the dataset and the effectiveness of the proposed method.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n+ The task of activity parsing is important and is likely to be the next step of video understanding beyond the current popular tasks of action recognition and detection.\n+ The dataset is new and well designed. It is a good supplement to existing datasets, mostly on action recognition and localization, and will benefit the computer vision community.\n+ The paper tries to give definitions of the subjects of study (activity, sub-activity, atomic action).\n+ The use of hyper-graphs to solve this problem is new and makes sense.\n+ Several tasks are defined and evaluated on this new dataset.\n\nWeaknesses\n\n- Parsing an activity into a hierarchy of sub-activities and atomic actions with spatial-temporal human-object interactions is not new. Several prior works on this direction are ignored. The authors need to credit those pioneering works and compare with them on the task and datasets.\n\n[1*] Wei, Ping, Yibiao Zhao, Nanning Zheng, and Song-Chun Zhu. \"Modeling 4d human-object interactions for event and object recognition.\" ICCV 2013.\n\n[2*] Pei, Mingtao, Yunde Jia, and Song-Chun Zhu. \"Parsing video events with goal inference and intent prediction.\" ICCV 2011.\n\n[3*] Wei, Ping, Yang Liu, Tianmin Shu, Nanning Zheng, and Song-Chun Zhu. \"Where and why are they looking? jointly inferring human attention and intentions in complex tasks.\" CVPR 2018.\n\n- I feel the tasks in Experiments fail to make full use of the dataset and diverge from 'activity parsing'. The input is a trimmed video with a single sub-activity segmented from the complete video. Why not use an input video at the activity level? If a sub-activity clip is the input, the task should be called sub-activity parsing instead of activity parsing. \n\n- Line 234. It is fine an existing method is used for activity parsing. But, for self-completeness, it is better to briefly introduce what the complex equation, including a lot of matrix multiplications, is doing. Also, H seems not defined.\n\n- Since the relationships among humans and objects are not known in advance, I wonder how the spatial-temporal hyper-graph is constructed. Will considering high-order relationships bring a large number of edges (for example, via enumerating)?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations and potential negative societal impact are well addressed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4 hours"}, "extra_text": {}}, {"review_id": "B-sP5wnEcaj", "reviewer": "Reviewer_Etdz", "summary": "The paper defined a new action representation called action hypergraph which connects multiple entities with hyperedges, the role classification for actors, and the  temporal tracking for entities. A new dataset (MOMA) following the defined representation is proposed. The paper also proposed a HyperGraph Activity Parsing (HGAP) network for tackling multiple activity understanding tasks simultaneously, e.g. activity recognition, sub-activity recognition, atomic action recognition, etc. Results for these activity recognition tasks, as well as the hypergraph construction evaluation results (hyperedge classification and role classification) are reported on MOMA, together with some simple baseline results.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The definition of action hypergraph is technically sound, since each action may involve multiple objects, and the action role and temporal tracking are also important for action understanding. The paper also proposes a new multi-task framework \"HGAP network\", and reports the results for each task separately. However, the results on Table 2 are hard to differentiate whether the result improvement comes from the action hypergraph definition with hyperedge, or is due to the proposed multi-task model \"HGAP network\". The baseline \"GCN [31]\" is modified from the Graph Convolutional Network (GCN) model [31] where hyperedges are decomposed to pair-wise ones. It is better to change hyperedge to pairwise edge connection in \"HGAP network\" to show the advantage of action hypergraph definition.\n\n2. All the results in this paper are built on top of GT bounding boxes (without class labels) and GT tracklets. Results using predicted bounding box proposals instead of GT bounding boxes should be reported, as is common practice in relevant tasks, e.g. dense captioning, scene graph generation, etc.\n\n3. For the role classification performance of proposed HGAP network in Table 4, it is compared with the baseline model ResNet-50 (37.9% mAP). However, another baseline model \"GCN [31]\" in Table2 reports a role classification accuracy of 64.7% mAP, which is significantly higher than both the baseline ResNet-50 (37.9% mAP) and proposed HGAP model (46.4% mAP) results on Table 4. Though the author has some explanation for this low baseline result in Table 4, but the explanation is not convincing. \".........To emphasize the importance of visual relationships, this baseline does not use graph structure on purpose..........\"\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "More extensive evaluation should be conducted to tackle the concern in the above \"Main Review\" section.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "30mm1oJF2D4", "reviewer": "Reviewer_QrDE", "summary": "This paper introduces Activity Parsing for the understanding of complex human activities based on a hierarchical structure describing human actions at 4 different levels. As an improvement from traditional pairwise entity representations, the Action Hyperedge is used to represent the more complex relationships between multiple humans and objects. Also the Action Hypergraph provides better interpretability for reasoning of high-level actions or activities. Followed by the concept of the hierarchical Activity Parsing, a new, meticulously-annotated video dataset, MOMA, is released and an HGAP network is proposed. In the experiment, the HGAP network achieves comparable results with the state-of-the-arts in several tasks such as video and role classification. \n", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes a novel hierarchy for parsing human activities, annotates a dataset based on the concept, and builds a network for activity parsing, and role/object classification. Works including video datasets for human activity recognition, action parsing methods, object detection, and graph neural networks are adequately cited. \n\nIt resolves the limitation in prior works in understanding complex, multi-person, multi-object human activities, and points out a new direction for video parsing. The release of the dataset will surely benefit the research in video understanding and human action detection/recognition. The proposed HGAP network takes advantage of the activity structure and combines the techniques in graph neural networks with video encoding networks for activity parsing and some downstream related tasks.\n\nTechnical soundness:\n\nThe design of the activity hierarchy and the HGAP network is quite reasonable. The authors consider the activity parsing as a set of several graph/edge/node classification problems and show in the experiments that the HGAP network outperforms the baselines on high-level tasks. However, at graph-level tasks, GCN outscores HGAP by a larger margin (Table 2). It’s not accurate to claim HGAP outperforms GCN in the abstract. \nIt is also surprising that GCN even outperforms HGAP Oracle at graph-level tasks. Ideally the ground truth hypergraph can be decomposed into pairwise graphs, so HGAP Oracle should be easier to train than GCN for both graph-level and high-level tasks. Please provide further explanations and deeper discussions for this.\n\nClarity of Expositon:\n\nOverall the clarity for the activity hierarchy and the dataset is good. Clear examples and illustrations (ex. the dining service) are used for explanation. However, there are some remaining questions.\n1. I agree the design of hyperedges is necessary because they better represent high-order actions, but the explanation for your hypergraph over the merged hypergraph from pairwise edges is not clear. Is this reason correct -- the two connections, (waiter, looking at, glass) and (customers, looking at, glass) cannot be merged because they might belong to different sub-activities? Please explain.\n2. It’s not very clear why the HGAP network is designed with several readouts until readers go through section 5. Some readouts are only used for specific tasks, so it would be better if illustrations of how the different levels of readouts are connected to the tasks in 5.1 can be shown in Figure 4. \n3. It remains unclear where the hypergraphs come from in the training process, since some components of them are the outputs of some tasks (ex. edge classification). The HGAP Oracle setup uses the ground truth graph structure. What about HGAP? Is the graph structure learned in the training process? If yes, how does it influence the classification results? \nAlso, are different weights used for different edge types in graph neural operations?\n \nSome minor errors:\n1. Figure 3 is not mentioned in the paragraphs.\n2. In line 209, it should be to^(no).\n3. In line 237, the term ROI shows up for the first time. Please provide its full form.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above for discussion on limitations of work.\n\nThe authors should elaborate on the discussions of the benefits/tradeoffs of the HGAP network. \n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2-3 hours"}, "extra_text": {"ethical_concerns": "The MOMA dataset is collected and annotated from public videos. The authors make a large effort to mitigate the ethical concerns in section 3.1. The dataset is not accessible during the review process, and careful checks should be carried out before the release.\n"}}]

Author rebuttal:

Meta-review: The authors introduce action hypergraph and an associated task (activity parsing), as well as a new dataset (MOMA) for complex activity recognition.  All of the reviewers are positive, and find the representation has certain novelty and the dataset will be valuable to the community.  The AC concurs that this paper is acceptable, and recommends a poster.

Decision: Accept (Poster)