Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a novel HOI detection pipeline which includes a human-object pair decoder and interaction decoder once visual feature is extracted. Both the human-object pair decoder and interaction decoder are modeled by a transformer architecture composed of multiple layers of attention. It also presents some techniques for further improvement such as dynamic re-weighting and pair-wise NMS (PNMS). The proposed method presents superior detection results on common benchmarks, namely HICO-Det and V-COCO.
main_review: Pros:

+ The analysis and comparison between two-stage and one-stage HOI detector is clear and reasonable. This section greatly motivates the proposed method.

+ The proposed method presents superior detection results on common benchmarks, namely HICO-Det and V-COCO. The improvement on rare classes is the most significant.

Cons:

- The title does not clearly convey the main message of the article.

-  What is "co-attention" is L148 and L166? Is it cross-attention?

- The reweighing formula is not clear. Does the superscript "(o, a)" mean either o(object) or a(action)? It will be better to focus on one at a time especially when formally presenting equation. Also, I don't understand what is "a query Q_o with length L_Q^o" and how is it integrated into the whole pipeline.

- Number of parameters and computation cost introduced by the transformer decoder?

- In Table 1, I think it is more suitable to list the results of CDN-S w/o re-weighing and PNMS for fair comparison with QPIC.
limitations_and_societal_impact: The authors discussed the potential negative societal impact of the work. However, I didn't see the limitations being discussed.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper explored the HOI detection problem and aimed to mine the benefits of two-stage and one-stage HOI detection methods. To this end, this paper firstly gave a detailed analysis of conventional one-stage and two-stage HOI detection methods and concluded the advantages and disadvantages of the conventional methods. Then, based on their analysis, this paper proposed a Cascade Disentangling Network (CDN), which is a one-stage framework with disentangling human-object detection and interaction classification in a cascade manner. Finally, this paper conducted experiments on HICO-Det and V-COCO datasets and achieved state-of-the-art performance.
main_review: + The motivation and the proposed method are clear and reasonable. With the development of one-stage and two-stage HOI detection methods, mining their advantages is the next essential step for the community.

+ This paper introduced a complete and accurate analysis about conventional HOI detection methods, and proposed a simple yet efficient solution to mine the benefits of one-stage and two-stage detector. The proposed method has achieved SOTA on two HOI detection benchmarks.

+ The results in Table 1 show that HO-PD makes a significant improvement (mAP from 15.37 to 24.05), which confirms the author's claim that one-stage methods are superior in human-object pair generation. I believe this will give some useful inspiration to the HOI field.

+ The source code is provided.

- The Dynamic Re-weighting process seems a bit complicated, hope to get a more easy-to-understand description.


limitations_and_societal_impact: yes
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: Inspired by its success on various vision and NLP tasks, Transformer based architectures have recently been adopted and achieved impressive performance on the task of HOI detection [3, 27, 38]. This paper builds upon these work and further innovates on the decoder architecture. The idea is that instead of having a single stage decoder which generates human and object bounding boxes and object and interaction class labels altogether, the decoding process is divided into two stages, where a first decoder is used to generate interactive human-object pairs, followed by a second decoder that predicts the interaction class. This resembles the traditional two stage HOI detectors which first detect individual objects followed by classifying the category of each pair. However, thanks to the development of Transformer based methods [3, 27, 28], one can directly predict human-object pairs in one go rather than relying on individual object detection and pairing. Meanwhile, the proposed framework can still build upon the same intuition which is to decouple interaction detection and classification so the network can just focus on one task at once. Besides this main architectural change, the proposed approach also features some new techniques on training (Sec. 3.4 "dynamic re-weighting") and post processing (Sec. 3.5 "pair-wise non-maximal suppression"). The approach is evaluated on standard datasets (HICO-DET and V-COCO) and has achieved notable gains over SOTAs in detection mAP.
main_review: ### Strengths
- The proposed approach is novel and sound. The two stage pipeline (i.e. proposal - classification) has already shown wide success in the object detection domain (e.g. Faster R-CNN). Many two stage approaches to HOI detection have been proposed to follow this paradigm but their method for proposal generation (i.e. detecting individual objects and pairing them) still seem naive and brittle. The approach proposed in this paper fills in this gap and provides a more elegant solution (by generating pair proposals in one go) using a Transformer architecture. In this sense, the proposed approach can be viewed as a Faster R-CNN counterpart for HOI detection.
- The benchmark results are remarkable. The paper has reported significant boosts in mAP against SOTA methods on both datasets (Tab. 2 and 3). E.g., from 29.07 to 32.07 on HICO-DET (Tab. 2) and from 58.8 to 63.91 on V-COCO (Tab. 3). This is impressive given the popularity of these datasets in publications recently.
- The writing has done a good job in introducing the background and motivating the proposed approach.
- Ablation is provided for justifying various new components in the pipeline.

### Weaknesses
- Clarity in Sec. 3.4 ("dynamics re-weighting") should be improved. The technical details are difficult to follow. For example, there are two queues (L177-179) (one for object and one for action), but there is only one set of weights in Eq. (4). Also there seems to be some connections from these weights to the loss but that connection is unclear. Finally, it will be helpful to first explain at an intuitive level how these weights are used in training.
- Missing details on the formulation of various training losses (i.e. $L_b$, $L_{GIoU}$, $L_p$, $L^o_c$, and $L^a_c$) in Sec 3.5. This piece of information is important for reproducibility and thus cannot be omitted.
- Following the last point, how are the ground truths of interaction scores defined (for $L_p$ in L197)? 

### Suggestions:
- [L155-157] "... to predict human bounding-box, object bounding box, and object class, ..." -> Besides these, there is also an "interactive score" in the output (See. Fig. 2 middle)? Please also add some details for that here.
- [L206] "instance classification" -> "object classification"
- [L207] "interactive probability" -> Consider using a consistent term ("interactive score" in Fig. 2 and L203) to minimize confusion. Also L197: "interactive probability loss".
- [L297] "10.08% and 9.32% point mAP gains" -> should be "relative mAP gains"?
- [Tab. 2 and 3] For the completeness of SOTA comparison, please add the benchmark results from [14] and [30] to Tab. 2 and 3.
- I am aware that CVPR'21 papers are officially published after the NeurIPS'21 deadline. But just a suggestion to add [1] below to the references in the next revision due to its high relevance to the technique herein. Also add their results to the SOTA comparison in Tab. 2 and 3.
- [Tab. 2] Please add a column for indicating whether the object detector has been fine-tuned on HICO-DET. For example, see Kim et al. ([1] below), [38], and [16].
- [Sec 4.5] Please add a pointer to Tab. 4 in the text.

[1] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J. Kim. HOTR: End-to-End Human-Object Interaction Detection with Transformers. In CVPR, 2021.

### Justification of Rating
The paper has made solid technical contributions with a novel Transformer based architecture. The experimental results also show strong promise. Although there are some issues on clarity and missing information, this is still overall a solid submission.
limitations_and_societal_impact: Yes, the authors have discussed the potential impact of their work in a dedicated section after the conclusion.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper explores the difference between the one- and two-stage based HOI detection methods and a corresponding method to conduct the human-object interactive pair detection and interaction classification separately, based on the Transformer structure and previous work QPIC. Moreover, two distinct methods named dynamic re-weighting and PNMS are also proposed. In experiments, the proposed model achieves impressive performance and outperforms its predecessor, QPIC.

Even the performance is good, comparing to QPIC, the proposed method is limited in design novelty. Moreover, some experiment details, especially the core separated pair detection and interaction classification performance contribution, are not verified. 

Overall, I think this work is interesting but still has a gap to achieve the acceptance bar of NeurIPS. Detailed comments are given below, and looking forward to the authors' response.
main_review: 1. Originality: 
The proposed model is basically based on QPIC. The two decoders are different. However, the performance gap between the two methods is small if removing the post-processing and re-weighting from the experiments. Thus, I think the novelty is marginal.

2. Quality: 
The implementation and experiments are reasonably good. The discussion about the one- and two-stage methods are clear and insightful.

3. Clarity: 
The model part is good, but some experiment details, especially the contributions of hyperparameters and strategies, are not clear. This makes me concern about the actual performance contribution of the core novelty, i.e., the separated pair detection and interaction classification. The code is given.

4. Significance: 
This paper has given some good points and analysis in terms of the discussion about the one- and two- stages.
In terms of the model design, the novelty is not so significant to afford the claim.
Regarding the experiment, I think some necessary tests are missing, e.g., the detailed interactive pair detection quality comparison, which is the core contribution. Simple HOI mAP cannot give enough hints and cues. 

Detailed comments:

Pros:
+ Good discussion about the difference between one-stage and two-stage methods. This is useful for the HOI community.

+ Code and method description make the readers easy to follow.

+ Impressive performance.

Cons:
- Though the proposed method is indeed different from previous methods, but is still a kind of two-stage in my opinion. In L277-279, the authors think the proposed method is one-stage. Nevertheless, from the previous discussion, I think CDN is more like a two-stage method. The difference is depended on the h-o pairing process. For example, TIN [17] also has the neg pair filtering process (NIS). However, this point does not affect the contribution.
- In training, the two decoders are trained together. So this makes the claim unaffected (L59) confusing and may cause misleading. Moreover, I think it would also make the disentangling claim in the intro weird. I suggest the authors better clarify this. Because after reading the intro, readers may think the h-o pair detection and interaction classification are *totally* separated in both training and inference. This is important and also related to the above naming of one or two stages. I think the authors think the difference lies at the inference stage.
- L79: M*N, this is, in fact, not very accurate. In HICO-DET, human-human interactions are also considered. Thus, the pairs can be M*(M+N).  This should be clarified in both method explanation and implementation.
- The interaction score looks weird. This is not mentioned in the intro but actually may play an important role in training and inference. First, what is the difference between it and interactiveness defined in TIN [17]? They seem to be the same and should be discussed. Moreover, how the supervision of the binary classification (a pair is interactive or non-interactive) affect the performance? Do the binary labels convert from the HOI labels?
- L88-91: the discussion about the feature of one-stage is interesting, but from current results, if SOTA two-stage methods adopt the detected boxes from the finetuned detector (on HICO-DET) or transformer, their performances are pretty close with recent one-stage methods. So what kind of feature is suitable for the interaction understanding? Please give some insightful discussion or give a small test on this. 
- L170, "each step can aggregate more related features to concentrate on its corresponding task", this is also confusing because the two decoders are trained together, and their gradients are entangled. Given the black-box property, I think this needs more verification instead of the good performance only.
- The Decoupling Dynamic Re-weighting part is similar to the method of Frustratingly Simple Few-Shot Object Detection. Please discuss this.
- Sec. 3.4 and 3.5 look not significantly related to the main contribution. Moreover, their performance contributions are not very clear. Does the 30.90 mAP come from the -B or -L setting? If using Res-50, compared to the QPIC (29.07), the gap is about one mAP. However, if using Res-101, the gap is less, as the QPIC with res-101 is about 29.90 [27] (missing in Tab.2). From the ablation study, the post-processing and re-weighting affect the performance a lot. So how the exact performance improvement of the core disentangling contribution? Please clarify this.
- About nine hyperparameters, how would they affect the performance? The ablation study does not give all analyses. Moreover, how they are decided. In Tab. 4, are they tuned based on the test set?
- L213 and 187, two alphas. L187 is p?
- Tab.1: the h-o pair quality can be directly compared by the interaction binary classification and box IoU.
- Fig 3: the difference is not so evident for me. Please give some more discussion according to the figs? For example, for ride-horse, does QPIC attention look better?

--Post-review: Appreciate the response from the authors. After reading the reviews and responses, my most concerns were addressed. I raising the rating to 6. For more details, please refer to the below response.
limitations_and_societal_impact: Yes.
Method and experiment limitations are listed above.
review_title: null

Meta-review: This paper presents work on human-object interaction (HOI) detection.  The main contributions include an analysis of two-stage versus one-stage HOI detection methods and a multi-layer transformer architecture that achieves state of the art results.  The initial reviews pointed to concerns over clarity of some terminology and overall originality and significance of the contributions.  However, the reviewers unanimously recommended acceptance of the paper after the discussion period, based on the good empirical results and sensible architecture.
Decision: Accept (Poster)