Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper defined a new action representation called action hypergraph which connects multiple entities with hyperedges, the role classification for actors, and the  temporal tracking for entities. A new dataset (MOMA) following the defined representation is proposed. The paper also proposed a HyperGraph Activity Parsing (HGAP) network for tackling multiple activity understanding tasks simultaneously, e.g. activity recognition, sub-activity recognition, atomic action recognition, etc. Results for these activity recognition tasks, as well as the hypergraph construction evaluation results (hyperedge classification and role classification) are reported on MOMA, together with some simple baseline results.

main_review: 1. The definition of action hypergraph is technically sound, since each action may involve multiple objects, and the action role and temporal tracking are also important for action understanding. The paper also proposes a new multi-task framework "HGAP network", and reports the results for each task separately. However, the results on Table 2 are hard to differentiate whether the result improvement comes from the action hypergraph definition with hyperedge, or is due to the proposed multi-task model "HGAP network". The baseline "GCN [31]" is modified from the Graph Convolutional Network (GCN) model [31] where hyperedges are decomposed to pair-wise ones. It is better to change hyperedge to pairwise edge connection in "HGAP network" to show the advantage of action hypergraph definition.

2. All the results in this paper are built on top of GT bounding boxes (without class labels) and GT tracklets. Results using predicted bounding box proposals instead of GT bounding boxes should be reported, as is common practice in relevant tasks, e.g. dense captioning, scene graph generation, etc.

3. For the role classification performance of proposed HGAP network in Table 4, it is compared with the baseline model ResNet-50 (37.9% mAP). However, another baseline model "GCN [31]" in Table2 reports a role classification accuracy of 64.7% mAP, which is significantly higher than both the baseline ResNet-50 (37.9% mAP) and proposed HGAP model (46.4% mAP) results on Table 4. Though the author has some explanation for this low baseline result in Table 4, but the explanation is not convincing. ".........To emphasize the importance of visual relationships, this baseline does not use graph structure on purpose.........."

limitations_and_societal_impact: More extensive evaluation should be conducted to tackle the concern in the above "Main Review" section.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper introduces Activity Parsing for the understanding of complex human activities based on a hierarchical structure describing human actions at 4 different levels. As an improvement from traditional pairwise entity representations, the Action Hyperedge is used to represent the more complex relationships between multiple humans and objects. Also the Action Hypergraph provides better interpretability for reasoning of high-level actions or activities. Followed by the concept of the hierarchical Activity Parsing, a new, meticulously-annotated video dataset, MOMA, is released and an HGAP network is proposed. In the experiment, the HGAP network achieves comparable results with the state-of-the-arts in several tasks such as video and role classification. 

main_review: This paper proposes a novel hierarchy for parsing human activities, annotates a dataset based on the concept, and builds a network for activity parsing, and role/object classification. Works including video datasets for human activity recognition, action parsing methods, object detection, and graph neural networks are adequately cited. 

It resolves the limitation in prior works in understanding complex, multi-person, multi-object human activities, and points out a new direction for video parsing. The release of the dataset will surely benefit the research in video understanding and human action detection/recognition. The proposed HGAP network takes advantage of the activity structure and combines the techniques in graph neural networks with video encoding networks for activity parsing and some downstream related tasks.

Technical soundness:

The design of the activity hierarchy and the HGAP network is quite reasonable. The authors consider the activity parsing as a set of several graph/edge/node classification problems and show in the experiments that the HGAP network outperforms the baselines on high-level tasks. However, at graph-level tasks, GCN outscores HGAP by a larger margin (Table 2). It’s not accurate to claim HGAP outperforms GCN in the abstract. 
It is also surprising that GCN even outperforms HGAP Oracle at graph-level tasks. Ideally the ground truth hypergraph can be decomposed into pairwise graphs, so HGAP Oracle should be easier to train than GCN for both graph-level and high-level tasks. Please provide further explanations and deeper discussions for this.

Clarity of Expositon:

Overall the clarity for the activity hierarchy and the dataset is good. Clear examples and illustrations (ex. the dining service) are used for explanation. However, there are some remaining questions.
1. I agree the design of hyperedges is necessary because they better represent high-order actions, but the explanation for your hypergraph over the merged hypergraph from pairwise edges is not clear. Is this reason correct -- the two connections, (waiter, looking at, glass) and (customers, looking at, glass) cannot be merged because they might belong to different sub-activities? Please explain.
2. It’s not very clear why the HGAP network is designed with several readouts until readers go through section 5. Some readouts are only used for specific tasks, so it would be better if illustrations of how the different levels of readouts are connected to the tasks in 5.1 can be shown in Figure 4. 
3. It remains unclear where the hypergraphs come from in the training process, since some components of them are the outputs of some tasks (ex. edge classification). The HGAP Oracle setup uses the ground truth graph structure. What about HGAP? Is the graph structure learned in the training process? If yes, how does it influence the classification results? 
Also, are different weights used for different edge types in graph neural operations?
 
Some minor errors:
1. Figure 3 is not mentioned in the paragraphs.
2. In line 209, it should be to^(no).
3. In line 237, the term ROI shows up for the first time. Please provide its full form.



limitations_and_societal_impact: See above for discussion on limitations of work.

The authors should elaborate on the discussions of the benefits/tradeoffs of the HGAP network.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes that video understanding be approached by trying to recognize at several different levels of granularity, simultaneously. At the highest temporal granularity is the "activity", which consists of a sequence of temporally-localized "sub-activities". Below that are the "atomic actions" performed by individual actors, and at the bottom an "action hypergraph" that is a scene hypergraph showing the relationships between actors & objects at the frame level. 
The authors introduce a video dataset, MOMA, and a task "Activity Parsing" which aims to infer all of these annotations from a video together with ground truth person and object tracks. Further they propose a HyperGraph Action Parsing Network, and compare it to 3DCNNs and graph convolutional networks on the Activity Parsing task.

main_review: [Originality] Although there have been a number of recent proposals formulating "temporal scene graphs for video", this paper provides a new perspective that I think is valuable, in particular because it unifies several video tasks -- classification, action localization, trimmed clip classification, spatiotemporal atomic actions, and visual relationships -- all under a single umbrella and embodied in a single dataset.

[Quality] The paper is generally very good. In particular the dataset represents a significant investment and seems valuable to the community. The HGAP model seems not particularly compelling due to uneven performance on the task, but that's fine for a dataset baseline. However my main question is around the ability to infer hypergraphs from video. For the technique to be useful, ultimately they will need to be inferred from raw, unannotated video, but in the paper's experiments, the graphs are given as input. Realizing the many degrees of freedom a hypergraph has, how reasonable is it that we can expect a model to be able to infer them from raw data? If inferring reasonable hypergraphs is very hard, then the modeling approach falls down. It would be nice to see how sensitive the approach is to errors in inferred hypergraphs; initial results in Table 2 (Oracle vs. non-Oracle) suggest that it might be very sensitive.

Similarly, it would be nice to see how reliably human annotators can produce a consistent, agreed-upon hypergraph for a video. This would be a good upper-bound for how well we think a machine could do it -- if people can't agree on the correct hypergraph, how could a machine do it?

[Clarity] The paper is clearly written, and generally well organized. However it's a little confusing that the experimental section is divided into three tasks. Upon digging deeper it's apparent that two of the tasks--5.2 Hierarchical Action Classification and 5.3 Role classification--are simply subtasks of 5.1 Activity Parsing, so it could just be a single task.

Another point that could be more clear is that the actual hypergraph structures are provided as input to HGAP / GCN, and not inferred directly from the input videos. I realized this only after seeing "action hypergraph generation from video" listed as future work in the appendix.

[Significance] I think the dataset is particularly valuable since connects coarse activities, to atomic actions, and to video scene graphs, which could enable research into more "holisitic" video understanding methods. Although its size is currently moderate, the authors note that it will soon contain 5x more annotated data, which will be impressive. Judged mainly on the significance of the dataset, I think this paper is valuable.

limitations_and_societal_impact: Not addressed. Any video action understanding model can be applied to the task of surveillance, although the labels introduced in this dataset seem benign.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper introduces a new dataset for video-based activity parsing. Instead of assigning a single action label to a video, the paper decomposes an activity into sub-activities and atomic actions. Instance-level annotations of actors, objects and their relationships are also annotated. Based on this dataset, the paper proposes a new model, action hypergraph, for activity parsing. Experimental results demonstrate the potential applications of the dataset and the effectiveness of the proposed method.
main_review: Strengths

+ The task of activity parsing is important and is likely to be the next step of video understanding beyond the current popular tasks of action recognition and detection.
+ The dataset is new and well designed. It is a good supplement to existing datasets, mostly on action recognition and localization, and will benefit the computer vision community.
+ The paper tries to give definitions of the subjects of study (activity, sub-activity, atomic action).
+ The use of hyper-graphs to solve this problem is new and makes sense.
+ Several tasks are defined and evaluated on this new dataset.

Weaknesses

- Parsing an activity into a hierarchy of sub-activities and atomic actions with spatial-temporal human-object interactions is not new. Several prior works on this direction are ignored. The authors need to credit those pioneering works and compare with them on the task and datasets.

[1*] Wei, Ping, Yibiao Zhao, Nanning Zheng, and Song-Chun Zhu. "Modeling 4d human-object interactions for event and object recognition." ICCV 2013.

[2*] Pei, Mingtao, Yunde Jia, and Song-Chun Zhu. "Parsing video events with goal inference and intent prediction." ICCV 2011.

[3*] Wei, Ping, Yang Liu, Tianmin Shu, Nanning Zheng, and Song-Chun Zhu. "Where and why are they looking? jointly inferring human attention and intentions in complex tasks." CVPR 2018.

- I feel the tasks in Experiments fail to make full use of the dataset and diverge from 'activity parsing'. The input is a trimmed video with a single sub-activity segmented from the complete video. Why not use an input video at the activity level? If a sub-activity clip is the input, the task should be called sub-activity parsing instead of activity parsing. 

- Line 234. It is fine an existing method is used for activity parsing. But, for self-completeness, it is better to briefly introduce what the complex equation, including a lot of matrix multiplications, is doing. Also, H seems not defined.

- Since the relationships among humans and objects are not known in advance, I wonder how the spatial-temporal hyper-graph is constructed. Will considering high-order relationships bring a large number of edges (for example, via enumerating)?
limitations_and_societal_impact: Limitations and potential negative societal impact are well addressed.
review_title: null

Meta-review: The authors introduce action hypergraph and an associated task (activity parsing), as well as a new dataset (MOMA) for complex activity recognition.  All of the reviewers are positive, and find the representation has certain novelty and the dataset will be valuable to the community.  The AC concurs that this paper is acceptable, and recommends a poster.

Decision: Accept (Poster)