Mining the Benefits of Two-stage and  One-stage HOI Detection
Authors: Aixi Zhang
Yue Liao
Si Liu
Miao Lu
Yongliang Wang
Chen Gao
XIAOBO LI
Venue: NeurIPS 2021 Poster (2021)
Keywords: Human-object Interaction Detection

Abstract: Two-stage methods have dominated Human-Object Interaction~(HOI) detection for several years. Recently, one-stage HOI detection methods have become popular. In this paper, we aim to explore the essential pros and cons of two-stage and one-stage methods. With this as the goal, we find that conventional two-stage methods mainly suffer from positioning positive interactive human-object pairs, while one-stage methods are challenging to make an appropriate trade-off on multi-task learning, \emph{i.e.}, object detection, and interaction classification.  Therefore, a core problem is how to take the essence and discard the dregs from the conventional two types of methods. To this end, we propose a novel one-stage framework with disentangling human-object detection and interaction classification in a cascade manner. In detail, we first design a human-object pair generator based on a state-of-the-art one-stage HOI detector by removing the interaction classification module or head and then design a relatively isolated interaction classifier to classify each human-object pair. Two cascade decoders in our proposed framework can focus on one specific task, detection or interaction classification. In terms of the specific implementation, we adopt a transformer-based HOI detector as our base model. The newly introduced disentangling paradigm outperforms existing methods by a large margin, with a significant relative mAP gain of 9.32% on HICO-Det. The source codes are available at https://github.com/YueLiao/CDN.

Mining the Benefits of Two-stage and One-stage HOI Detection
Aixi Zhang'*
Yue Liao Si Liu
Miao Lu
Yongliang Wang'
Chen Gao Xiaobo Li'
Alibaba Group
"Beihang University
Abstract
Two-stage methods have dominated Human-Object Interaction (HOI) detection for several years. Recently, one-stage HOI detection methods have become popular. this paper, we aim to explore the essential pros and cons of two-stage and one-stage methods. With this as the goal, we find that conventional two-stage methods mainly suffer from positioning positive interactive human-object pairs, while one-stage methods are challenging to make an appropriate trade-off on multi-task learning, ie.. object detection, and interaction classification. Therefore, a core problem is how take the essence and discard the dregs from the conventional two types of methods. To this end, we propose novel one-stage framework with disentangling human-object detection and interaction classification a cascade manner. detail, we first design a human-object pair generator based on state-of-ther the-ttr-rtt one-stage HOI detector by removing the interaction classification module or head and then design relatively isolated interaction classifier to classify each human-objec pair. Two cascade decoders in our proposed framework can focus on one specific task, detection interaction classification. In terms of the specific implementation, adopt a transformer-baat HOI detector as our base model. The newly introduced disentangling paradigm outperforms existing methods by large margin, with a significant relative mAP gain of 9.32% on HICO-Det. The source codes are available https: / httptttttttttt//t////////t com/ /ueLiaa/CDN..
1 Introduction
The goal Human- Object Interaction (HOI) detection [2, 20, 6, 18, 8, 17,3] is to make a machine detailedly understand human activities from static image. Human activities in this task abstracted as a set of <human, object, action> HOI triplets. Thus, an HOI detector is required to locate human- object pairs and classify their corresponding action simultaneously. Based on this definition, can summarize conventional HOI detection methods into two paradigms, i.e., two-stage methods, and one-stage methods. These two paradigms have made significant progress with the development of deep learning, but both paradigms still have their shortcomings due to their structural design. This paper aims to present detailed analysis of methods under these two paradigms and propose solution to mine the benefits of two-stage and one-stage methods.
We first take closer look the conventional two-stage and one-stage HOI detectors. Conventional two-stage methods [6, 2, 18, 5] are mostly with a serial architecture. As shown in Figure (a), two-stage methods detect humans and objects first and then feeds the human-object pairs, which are generated by matching humans and objects one by one, into an interaction classifier. The serial architecture suffers from locating the interactive human-object pairs under the interference of large number of negative pairs only based on local region features. Otherwise, the efficiency of
"quual contribution Corresponding author (liusi @ buaa.cdu.cn)
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Instance Drlector
"how'
Insuene Desetion
Clasaficction Oue
Malii-ze Leariing
with
Inturation Choopiiation Asoxckeiun
Hlsransbjaa Pair Generatee
Chasification
(4). Two-stage framework
(b). One stage end to end framework
(e). One-stage frimework with parallel anchitucture
(d). Our ene stage framework with cascade disentangling decoder
Figure 1: (a) Two-stage framework, (b) one-stage end-to-end framework, (c) one-stage framework with parallel architecture, and (d) our one-stage framework with a cascade disentangling head.
two-stage methods is also limited by the serial architecture. To alleviate these problems, one-stage methods [20, 13, 39, 3, 28, 14] are proposed to detect the HOI triplets directly and break HOI detection as multi-task learning, i.e., human-object detection and interaction classification, which is shown in Figure 1(b). Therefore, one-stage methods can easily focus on the interactive human -ojjcct pairs and effectively extract corresponding features in end-to-end manner. However, it difficult for single model to make a good trade-off on multi-task learning since human-object detection and interaction classification are two very different tasks, which requires the model focus on different visual features. As shown in Figure (c)), though some previous methods [20, 3] design two parallel branches to detect instances and predict interaction respectively, the interaction classification branch still needs to regress additional offsets to associate humans and objects. Thus the interaction branch is also required to make a trade-off between interaction classification and human and object positioning.
Therefore, the intuitive idea is to take the essence and discard the dregs from the two paradigms. To attain this, we propose a novel end-to-end one-stage framework with disentangling human-object detection and interaction classification in cascade manner, namely Cascade Disentangling Net- work (CDN). The original intention of our framework is to keep the advantages of conventional one- stage methods, directly and accurately locating the interactive human-object pairs, and bring the advantages of two-stage methods into our one-stage framework, disentangling human-object detection and interaction classification. As shown in Figure Idd), in our proposed framework, we design human-object pair decoder based on the one-stage paradigm by removing the interaction classification function, namely HO-PD, and following an isolated interaction classifier. To instantiate our idea with an end-to-end manner, we design the HO-PD based on the previous state-of state-of-the-ard the one-stage transformer-basee HOI detector, HOI-Trans [39] and QPIC [28], where we remove the interaction classification head for each query and make focus on human-object pairs detection. Otherwise, we design independent HOI decoder for interaction classification make the interac- tion classification unaffected by human-object detection. Therefore, there exists a core problem, iee. how to link the human-objes pair and the corresponding action class. To address this problem, initialize the query embedding of the HOI decoder with the output of the last layer of the HO-PD. In this case, the HOI decoder is able learn the corresponding action category under the guidance of the query embedding and free out from the human-object detection task. Moreover, we design a decoupling dynamic re- reieeigh manner handle the long-tailed problems in HOI detection.
Our contributions can be summarized threefold: (1) We conduct detailed analysis of two conven- (2) We propose novel one-stage framework with a cascade disentangling decoder to combine the advantages of two-stage and one- stage methods. (3) Our method outperforms previous state-of-ther -fee------e-ttttt methods by large margin the HOI detection task, especially achieves a 25.35% performance gain on rare classes of HICO-Det.
tional HOI detection paradigms, i.e., two-stage and one-stage.
Analysis of Two-stage and One-stage HOI detectors
We first introduce a unified formulation for the HOI detection problem. Given a human-centric image I, the model (-() ) required predict a set of HOI triplets S {(t/: 69. az), E 21, KK) K}}, where 6/. by and a, denotes human bounding-box, an object bounding-box and their corresponding action category, respectively.
Two-stage HOI detector.
Two-stage detectors can be regarded as an instance-driven manner, detect- ing instances first and predicting interaction based on the detected instances. The two-stage detector divides T(-) into two stages, i.e., detection Ta(-) and interaction classification T-). In the first stage,

---

we suppose that Ta(-) produces M human bounding hounding-boxes boxes and N object bounding-boxer Here the 'object' is universal object which includes human as one class. Therefore. Ta(+) generates M N human-object pairs. In general, the number of true-positive interactive human-object pairs, denoted as K", is much smaller than M x N. However, the second stage, Tcl.) needs to scan all M x N pairs one by one and predict an action category with its corresponding confidence score. In this case, T.(-) is required to inference this manner causes three problems. Firstly, these models produce a more additional computational cost, whose time complexity is ( x N) > O(K"). Secondly, the imbalance between positive and negative samples makes the model easily overfit to negative samples. Thus the model tends to assign a 'no-interaction" class for human object pairs with very high confidence, suppressing the true- sue--ositive samples. Thirdly, the accuracy of interaction classification is influenced by the non-end-to-en pipeline. Because the interaction classification is mostly based on the region features extracted by Ta(:), while the core of (T(()) is to regress bounding-boxet and its extracted features pay more attention to the edge of regions, thereby such features are not good options for interaction classification, which needs more context. However, it is excellent property for two-stage methods that disentangling detection and interaction classification makes each stage focus on its task and produce good results in each stage.
x N times to find K' interactive pairs from M x N pairs. We argue that
One-stage HOI detector. As for one-stage methods, they detect all HOI triplets S directly and simultaneously with an end-to-end framework. Such paradigm has greatly relieved the three problems of two-stage methods, especially for efficiency, where the time complexity is reduced to O(K') Most one-stage methods interaction-driver or interactive human-object pairs [39], thereby reducing negative sample interference. However, coupling human-object detection and interaction classification limit their performance because it is hard to generate a unified feature representation for two very different tasks. Though the parallel one-stage methods break HOI detection into two parallel branches, their interaction branch still suffers from multi-task learning. Specifically, the optimization target of interaction branch is P(es, co, a) V), where eμ, and e, are associative embeddings, e.g. offset, to match interaction with human and object respectively. Therefore, even if detection is organized as an independent branch, the interaction branch must position humans and objects for the association. The set-based detectors couple detection and interaction completely, whose optimization function is P(b",, bb, a(V).
which directly locate the interaction point [20]
Next, introduce simple one-stage framework with disentangling human-object detection and interaction classification, namely CDN, to mine the benefits of two-stage and one-stage HOI detectors. Our CDN disentangles the original set-based one-stage optimization function into two cascade decoders. Firstly, we predict human-object pair by P(bh,b' V)) Secondly, we apply an isolated decoder to predict the action category by P(a V bh, bo). More details are in the following.
3 Method
In this section, we will present a detailed introduction to the pipeline of our proposed CDN, In section 3.1, we present an overview of our framework and briefly introduce the pipeline. In section 3.2, we introduce the visual feature extractor. The cascade disentangling HOI decoder is introduced in section 3.3. Section 3.4 introduces a novel dynamic re-weighting mechanism that mitigates the long-tailed problem. The detailed training process and post-processing are discussed in section 3.5.
3.1 Overview
The architecture of our proposed CDN is illustrated in Figure 2. Our CDN is organized in a cascade manner with a visual feature extractor. Given an image, we first follow transformer-bart based detection methods [1. 39] to apply CNN followed by transformer encoder architecture to extract visual features into sequence. Then we detect HOI triplets in two cascade decoders. Firstly, we apply the Human-Objed Pair Decoder (HO-PD) to predict set of human-object bounding- pairs based on a set of learnable queries. Next, taking the output of the last layer of HO-PD as queries, an the action category for each query. Finally, the HOI triplets are formed by the output of the above two cascade decoders.
isolated interaction decoder is utilized to predict
3.2 Visual Feature Extractor
We define the visual feature extractor by combining CNN and transformer encoder. Fed with an input image I with shape (H, W, C), the CNN generates feature map of shape (H' w',, Do).
3

---

Globel
Hunzn-objex Queics
Intenction Quaia
N
Qa
Qc
b N
X
Object
Internction
Transformer
Pair Decoder
Qa
Decoder
UNN
pos
X
Object
Adion category
Postional Fiooding
Flation Foxtire
Visuall
Interaction
Figure 2: The framework of our CDN. is comprised of three components. Visual Feature Extractor, Human-Object Pair Decoder (HO-PD) and Interaction Decoder. We first apply a CNN-transformer combined architecture to extract sequenced visual features X.. Then, we divide HOI detection into two cascade transformer-base pairs based on X, and set of random-initialize from a binary classification to determine whether the human-objec pair is an interactive pair or not. Secondly, predict one many action categories for each predicted human-object pairs, where take the output of HO-PD Q to initialize the interaction queries Qe and aggregate information with X,, s- Finally, the HOI triplets formed by the output of the cascade decoders.
decoders. Firstly, we regress the human-object bounding-ber
queries Qa by HO-PD. The interactive score is
Then, D3 is reduced to De by a projection convolution layer with kernel size x 1. Next, a flatten operator used to generating the flatten feature X, E R xw') Xx by collapsing the spatial dimensions into dimension, This flatten feature is then fed into transformer encoder and the position encoding Epss E R( xW >x,,, which distinguishes the relative position in the sequence >xD,, xw'XxD, Thanks the multi-head self-attention mechanism, the transformer encoder
X, E R(H xw" produces feature map with richer contextual information by summarizing global information. The output of the encoder is denoted as global memory with dimension of D
3.3 Cascade Disentangling HOI Decoder
The cascade disentangling HOI decoder consists of two decoders: Human Objecct Pair Decoder (HO- PD) and interaction decoder. Both decoders share the same architecture, a transformer-basee decoder, with independent weights. In this subsection, first introduce the general architecture of the decoder and then elaborate on the two decoders in detail, respectively.
Transformer-baseee decoder. We follow the transformer-based object detector DETR [1] to design basic architecture in our cascade disentangling HOI decoder. apply N transformer decoder layers for each decoder and equip each decoder layer with several FFN heads for intermediate supervision. Specifically, each decoder layer is comprised a self-attention module and a multi-head co-attention module. During feed-for forwadd, fed into set of learnable queries Q R NC,, each decoder layer first applies self-attention module on all queries and then conducts a multi- co-attention operation between queries and the sequenced visual features, and outputs queries. For the FFN heads, each head is comprised of one or several MLP branches, and each branch is for specific task, e.g., regression, classification. All queries share the same FFN heads. Therefore, each decoder can be simply represented as:
of updated
P f(Q, Xs, Epos)
Besides, the number of queries N, is determined by the number of positive samples of an image.
HO-PD. Firstly, we design the HO-PD to predict set of human-object pairs from the sequenced visual features. To this end, we first randomly initialize HO queries. Then we apply a transformer- visual features as input and applies three FFN heads for each query to predict human bounding- object bounding-box, and object class, which form human-object pair. We also utilize additional interactive score head to simply determine whether the human-object pair an interactive pair or not by binary classification. In this case, P is instantiated as Pho. which is consist of set of
of learnable queries Qu R Na xO as
decoder, which takes HO queries Qu and sequenced

---

human-object pairs {(6) i (1,2. -. NNa)). Thus, HO-PD can be denoted as:
Pro = fa(Qa, x,, Epos)
(2)
In addition, we keep the output queries of the last layer of HO-PD as Q2 for the following step.
Interaction Decoder. Secondly, we propose the interaction decoder to classify the human-object queries and assign one several action categories for each human-object query. To classify each human-object query one-to-one, we initialize Q. with the output of HO-PD Qa In this way, Q2tt can provide prior knowledge guide Qe to learn the corresponding action categories for each human-obye object query. The other components and inputs are the same as HO-PD, which conducts self-attention among queries and co-attention with X, and Epoz The final output is a of action categories Pas : {ai, (1,2, ... [N)) Therefore, this process can be formulated as:
Pas = (es X., Epos).
In our proposed cascade disentangling HOI decoder, the task of HOI detection is disentangled into two relatively independent steps: human-object pairs detection and interaction classification. Therefore, each step can aggregate more related features to concentrate on its corresponding task.
3.4 Decoupling Dynamic Re-weighting
The HOI datasets usually have long-tail class distribution for both object class and action class. To alleviate the long-tail problem, we design a dynamic re-weighting mechanism for further improve- ments with a decoupling training strategy. Then, we freeze the parameters of the visual feature extractor and only train the cascade disentangling decoders with a relatively small learning rate and the designed dynamic re- re-weighted losses.
detail, we first train the whole model with regular losses.
During decoupling training, at each iteration, we apply two similar queues to accumulate number of each object class or action class. The queues are used as memory banks to accumulate training samples and truncate the accumulation with length LQ as sliding windows. In detail, Qo with length La to accumulate object number N for each object class E {1, 2, ... C. and Qa with length T& to accumulate interaction number No for each action category i E {1, 2, .... C. The dynamic re- weightiitn coefficients Wdynomic presented as follow:
No
Po
Σ
N
i=
who
i=
(4)
(E(112,-....A
N₂
No
bg
Ziml No
Pa
Σ Liml
No
Po
0
(6(1,2......C.
N;
who
(5)
bg
where N, is the number of accumulated positive samples of category by the queues Qo and Qa. Nog is the number of accumulated background samples, C' is the number of categories, and exponent is hyper-parameter that adapts the magnitude of mitigation. Specifically, the weight of background class, Whg. is designed to balance the positives and negatives. For the stability of the dynamic re- weighted training, the weight coefficients are initialized as Wstatic with those calculated by 4 and 5 using the static number of object and action categories. The final dynamic weights are given as w WWatatie + (1 7))ddnmmies where y is smooth factor, given as min(0.999 0.9). The factor transits w from Wstatic to Wdynamic with the increasing of LQ. Finally, the weights are used to the classification losses traditional way by multiplying each coefficient to each class and then calculating the summation.
3.5 Training and Post-processing
In this section, introduce the training and inference processes in detail. Especially, we will introduce a novel Pair-wise Non-Maximal Suppression (PNMS) strategy in the inference process.
Training. Following the set- based training process of HOI-Trans [39] and QPIC [28], we first match each ground-truth with its best-matching prediction by the bipartite matching with the Hungarian algorithm. Then the loss is produced between the matched predictions and the corresponding ground truths for the final paok-propagttion. During matching, we consider the predictions of two cascade decoders together. The loss of CDN follows QPIC which is composed by five parts: the box regression
5

---

loss Lg. the intersection- peneecctiooover-umieee uver- loss LGIOU [26], the interactive score loss Lp. the object class loss Lg and the action category loss Lg The target loss is the weighted sum of these parts as:
=
(NoLi + )crat - Aplap + AoLi + daLes Σ (E().
(6)
where Ao. AGIOU, Ap. do and Aa are the hyper-parameters for adjusting the weights of cach loss.
Inference. The inference process is to composite the output of instance-related FFNs and the interaction-related FFN to form HOI triplets. By our cascade disentangling decoder architecture, the instance queries and the interaction queries are one-to-one corresponding, therefore, the five components <human bounding box, object bounding box, object class, interactive score, action class> can be homologous output prediction as <b/F, b2, argmax where of and ci are the scores of interaction and object classification, respectively, and ee is the interactive score from the interactive FFN head for the query vector being an human-object pair.
each of the N₂ dimensions per FFN head. Formally, we generate the i-th
The HOI triplet score c/ is given by ch c
PNMS. After sorting ch in descending order and generating the top K HOI triplets, we design pair-wise non-maximal suppression (PNMS) method to further filter out human-object pairs from pair-wise bounding boxes overlapping perspective. For two HOI triplets m and n, the pair-wise overlap PloU is calculated as:
T b PIoU Hom[m,n n) = (TC B4)
T 12) 3 bg)
(7)
(be
where the operators I and U compute the intersection and union areas between the two boxes of m and n, respectively. a and 3 are the balancing parameters between humans and objects.
Experiments
In this section, conduct comprehensive experiments demonstrate the superiority of our designed CDN. In section 1. we briefly introduce the experimental benchmarks. Section 4.2 presents implementation details. Next, It is detailed experimental comparison and analysis of two-stage and one-stage methods in section 4.3. In section 4.4, we compare our methods with the previous state-of-the-arr methods. The ablation studies and components analysis are included in 4.5.
4.1 Datasets and Evaluation Metries
Datasets. We carry out experiments on two widely-used HOI detection benchmarks: HICO-Det [2] and V-COCO [8]. We follow the standard evaluation scheme. HICO-Det consists of 47, 776 Creative Common images from Flickr (38, 118 for training and 9, 658 for test) with more than 150K human- object pairs. It contains the same 80 object categories MS-COCO [21] and 117 action categories. The objects and actions form 600 classes of HOI triplets. V-COCO is derived from MS-COCO dataset, which consists of 5, 400 images the trainval subset and 4, 946 images in the test subset. has 29 action categories (25 HOIs and 4 body motions) and 80 object categories. For both datasets, one person can interact with multiple objects in different ways at the same time.
Evaluation Metrics. Following the standard evaluation [2], we use the mean average precision (mAP) as the evaluation metric. For one positively predicted HOI triplet <human, object, action>, needs to contain accurate human and object locations (box IoU with reference to GT box is greater than 0.5) and correct object and action categories. Specifically, for HICO-Det, besides the full set of 600 HOI classes, we also report the mAP over rare set of 138 HOI classes that have less than 10 training instances and a non-rare set of the other 462 HOT classes. For V-COCO, we report the role mAP for two scenarios: scenario 1 includes the cases even without any objects (for the four action categories of body motions), and scenario 2 ignores these cases.
4.2 Implementation Details
We implement three variant architectures of CDN: CDN-S, CDN-B, and CDN-L, where 'S', 'B', and 'L' denote small, base, and large, respectively. For CDN-S and CDN-B, we adopt ResNet-50 with a 6-layer transformer encoder as the visual feature extractor. For the cascade decoders, CDN-S is equipped with both 3-layer transformers, while CDN-B 3 has a 6-layer transformer for each decoder. CDN-L only replaces the 5esNet-50 with ResNet-101 in CDN-B. The reduced dimension size D.

---

is set to 256. The number of queries Na is set to 64 for HICO-Det and 100 for V-COCO since the average number of positives for variant human-object pairs per image of HICO-Det smaller than V-COCO. The human and object box FFNs have 3 linear layers with ReLU, while the object and action category FFNs have one linear layer. The code is provided in supplemental material.
During training, we initialize the network with the parameters of DETR [1] trained with the MS- COCO dataset. We set the weight coefficients λb. AGIOU' Ap. A and X to 2.5, 1, 1, and 1, respectively, which are exactly same with QPIC [28]. We optimize the network by Adam [23] with the weight decay 10-4. We first train the whole model for 90 epochs with a learning rate of 10- decreased by 10 times at the 60th epoch. Then, during the decoupling training process, we fine-tune the cascade disentangling decoders together with the box, object, and action FFNs for 10 epochs with a learning rate of 10 5, We use both object and action dynamic re- re-weighting for BICO-Det and only action dynamic re- weighting for V-COCO. The re- re-weighting parameter p is set to 0.7 for both object and action. The length L of training sample queue Q for both object and action is set to 2 x N where N, is the sample number of the training set. All experiments are conducted on the 8 Tesla V100 GPUs and CUDA10.2, with a batch size of 16.
For validation, we select 100 detection results with the highest scores and then adopt PNMS to further filter results. The threshold, a, and / of PNMS are set to 0.7, 1, and 0.5, respectively.
4.3 Experiment Analysis of Two-stage and One-stage Methods
In this part, we introduce detailed experimental analysis of conventional two-stage and one-stage methods and our proposed CDN from the following three aspects.
Human-object Pair Generation. We first explore the quality of the human-object pairs generation between two-stage and ote-stage methods. To attain this, we conduct a detailed experiment based on representative two-stage method iCAN [6]. We first implement PyTorch version iCAN as the bascline model, denoted as iCAN', which only applics human and object appearance with a COCO-pretrained Faster-RCNN detector [25]. For fair comparison, first fine-tune DETR on HICO-Det for 100 epochs only with the instance detection annotation based on COCO-pretrained weights. Then combine the detected human and object bounding-boxee whose confidences are greater than a threshold, one by one to generate couman-objec pairs denoted as iCANí in Table 1. We train our CDN only with HO-PD for 100 epochs and get the human-object pairs from the output directly. Then, we graft the human-object pairs to the baseline model to extract box features and utilize the same interaction classifier in the second stage of iCAN" In this way, we degrade the number of pairs from M x N to K", which means time complexity is reduced from O(M x N) to OK')). Primarily, HO-PD significantly promotes mAp from 15.37 to 24.05, as shown in Table 1. This indicates that one-stage methods are much superior in human-object pair generation.
Strategy
Full
Rare
Non-Rare
hold cake
walk dog
ride bicyclu
ruw busl
ride honse
ICAN*
14.16
12.26
14.73
ICANT
15.37
13.23
16.01
(+)
HO. PD+CCAN
24.05
18.32
25.76
QPIC
29.07
21.85
31.23
CDN-S base
30.96
27.02
32.14
(b)
Table I: Analysis of Two-stage and One- stage Methods. denotes our implemented PyTorch version iCAN [6] baseline model. f denotes replacing instance detection boxes given by a HICO-Det fine-tuned DETR detector to extract box features. 'HO- PD+ICAN" denotes replacing original one- by-one generated human-object pairs with our HO-PD generated. *CDN-S base" de- notes CDN-S w/o re-weighting and PNMS strategies.
(c)
Figure 3: Visualization of Feature Maps for Queries. Visual attended features for query with top-1 score extracted from the last layer of the de- coder of (a) QPIC, (b) HO-PD in CDN, and (c) interaction decoder in CDN. Zoom in for details.
Interaction Classification. We aim to study the interaction classification between conventional multi-task one-stage methods and our disentangled one-stage detector. We can regard QPIC [28] as
7

---

Default
Know Object
Method
Detector
Backbone
Extra
Full
Rare
Non-Rare
Full
Rare
Noa-Rare
Two-stage Method:
InteractNet
COCO
ResNet-50-EPN
9.94
7.16
10.77
GPNN [24]
COCO
Res-DCN-152
K
13.11
9.34
14.23
ICAN [6]
COCO
ResNet-50
X
14.84
10.45
16.15
16.26
11.33
17.73
No-Frills
COCO
ResNct-152
P
17.18
12.17
18.68
PMFNet [30]
COCO
ResNet-50-FPN
P
17.46
15.65
18.00
20.34
17.47
21.20
CHGNNe [31]
COCO
ResNet-50
X 17.57
16.85
17.78
21.00
20.74
21.08
DRG
COCO
ResNet-50-FPN
T
19.26
17.74
19.71
23.40
21.75
23.89
VCL [12]
COCO
ResNet-550
K
19.43
16.55
20.29
22.00
19.09
22.87
IP-Net [32]
COCO
Hourglass-10k
X 19.56
12.79
21.58
22.05
15.77
23.92
VSGNet [29]
COCO
ResNet-152
20.91
FCMNet [22]
COCO
ResNet-50
K
20.41
17.34
21.56
22.04
18.97
23.12
ACP [15]
COCO
ResNet-152
T
20.59
15.92
21.98
[35]
COCO
ResNet-152
22.28
18.88
26.54
DJ-RN [16]
COCO
ResNet-50
P 21.34
18.53
22.18
23.69
20.64
24.60
IDN [17]
COCO
ResNet-50
X 23.36
22.47
23.63
26.43
25.01
26.85
One-stage Method:
UnionDet [13]
COCO
ResNet-50-FPP
X
17.58
11.72
19.33
19.76
14.68
21.27
DIRV [4]
COCO
EfficientDet-U d3
X 21.78
16.38
23.39
25.52
20.84
26.92
PPDM-Hourglass [20]
HICO-DET
Hourglass-Ie 104
X 21.94
13.97
24.32
24.81
17.09
27.12
HOI-Trans [39]
HICO-DET
ResNet-50
X
23,46
16.91
25.41
26.
22
GG GG-Net [37]
DICO-DET
Hourglass- 104
X
23.47
16.48
25.60
27.36
20.23
29.48
ATL [1J]
HICO-DET
ResNet-50
X
23.81
17.43
25.72
27.38
22.09
28.96
HOTR [14]
HICO-DET
ResNet-50
X
25.10
17.34
27.42
AS- [3]
HICO-DET
ResNet-50
X
28.87
24.25
30.25
31.74
27.07
33.14
QPIC [28]
HICO-DET
ResNet-50
X
29.07
21.85
31.23
31.68
24.14
33.93
CDN-S
HICO-DET
ResNet-50
31.44
27.39
32.64
34.09
29.63
35.42
CDN-B
HICO-DET
ResNet-50
X
31.78
27.55
33.05
34.53
29.73
35.96
CDN-L
HICO-DET
ResNet-101
X
32.07
27.19
33.53
34.79
29.48
36.38
Table 2: Performance comparison on the HICO-Det test set. The 'P', 'T' represent human pose information and the language feature, respectively.
multi-task version of our CDN. Table shows that our 'CDN-S base' (w/o we-weighting and PNMS strategies) has achieved mAP 30.96 with 6.50% relative mAP gain compared to QPIC. Especially, our "CDN-S base' significantly outperforms QPIC for rare classes with 23.66% improvement. The performance of rare classes can partly reflect the accuracy of interaction classification.
Feature Learning. This part discusses the differences in feature learning between the conventional one-stage method, QPIC, and our CDN from a qualitative view. As shown in Figure 3, we visualized the feature maps extracted from the last layer the decoder of QPIC, the HO-PD, and the interaction decoder in our CDN. We can see that HO-PD and QPIC attend very similar regions, e.g., the boundaries humans and objects and the human-object contact areas, which are beneficial for locating the interactive human-object pairs. However, the interaction decoder concentrates on human- pose and the regions that contribute to understanding human actions. As for the specific case, for example, for 'hold cake'. HO-PD in CDN attends to the boundaries of the cake while the interaction decoder in CDN concentrates on the interaction context, i.c., the human's 's hands holding the cake. Thus, it shows that CDN disentangles the human-object detection and interaction classification. For 'ride horse', HO-PD in CDN emphasizes the overall feature of the human and the horse, and the highlight parts are the edges of the human and horse. For the interaction decoder in CDN, the highlighted part emphasizes the interaction context, i.e., the human carries the rope when riding a horse. Finally, QPIC somehow combines the two highlights, but both not obvious.
4.4 Comparison to State-of-the-Art
We conduct experiments on HICO-Det and V-COCO benchmarks verify the effectiveness of our proposed methods. For HICO-Det dataset as shown in Table 2, comparing to the previous state-of-the- two-stage method FCMNet [22] with ResNet-50 backbone, our CDN-B significantly promotes mAP from 20.41 to 31.78, with relative gain of 55.71%. Even compared with PD-Net [36] which adopts extra language feature and DJ-RN [16] which utilizes extra human pose features, CDN-B achieves 52.77 22.71% % and 48.92% relative mAP gains, respectively. When comparing to the one-stage method AS-Net [3] and QPIC [28] which also adopt transformer-based detector architecture, CDN-B attains 10.08% and 9.32% point relative mAP gains, respectively. Table 3 shows the comparisons on V-COCO dataset. CDN-B achieves 62.20 APrate Prote on Scenario and 64.42 A APrate on Scenario which significantly outperform previous state-of-the- method QPIC with relative 5. 5.94% and 5.61%

---

Method
Extra AP
rale AP
rols
Two-stage Method:
Strtteyy
Full
Rare
Non-Rare
InteractNet [7]
X
40.0
QPIC [28]
29.07
21.85
31.23
GPNN [24]
X
44.0
base
31.06
26.68
32.36
ICAN [6]
X
45.3
52.4
+ re-weighting
31.38
27.36
32.58
TIN [18]
X
47.8
54.2
PNMS
31.78
27.55
33.05
VCL [12]
X
48.3
(a) Strategies: Analysis of improvements by various training strategies.
DRG [S]
51.0
IP-Net [32]
X
51.0
VSGNet [29]
X
51.8
57.0
Sinwegy base decouple
L.
Full
PMFNet [30]
P
52.0
31.06
26.68
32.36
PD-Net [35]
T
52.6
30.90
26.09
32.33
static
0.7
31.25
27.12
32.49
CHGNet [31]
X 52.7
dynamic
2 x N,
31.33
27.45
FCMNel [22]
53.1
dynumic
1 x N.
0.7
31.34
27.48
32.49
ACP [15]
T 53.23
dynamic
2 x N,
0.7
31,38
27.36
32.58
IDN [17]
X
53,3
60.3
(b) Dynamic re-weighting: Analysis decouple training with
Ote-stage Method:
dynamic weighted losses, če., different quene length lup, coeffi- cient and dynamic or static.
UnionDet [13]
X
47.5
56.2
HOI-Trans [39]
X
52,9
a
B
thres
Rare
Non-Rars
AS-Net [3]
X
53,9
31,38
27.36
32.58
1
0.8
31,66
27.46
32.91
GG-Net [37]
X
54.7
1
0.7
31.75
27.50
33.03
HOTR [14]
X 55.2
64.4
0.7
0.7
31.77
27.54
33.03
DIRV [4]
X 56.1
0.5
0.8
31.75
27.51
33.02
QPIC [28]
X
58.8
61.0
0.5
0.7
31.78
27.55
33.05
CDN-S
X 61.68
63.77
(c) PNMS: The effects different settings PNMS coef- ficients, a,, B, and fhres denotes threshold.
CDN-B
X 62.29
64.42
CDN-L
X
63.91 65.89
Table Ablation studies our proposed method on the HICO-Det test set. We carry out all experiments based on the base model (CDN-B).
Table 3: Performance comparison the V-COCO test set. The 'P', "T" rep- resent the human pose information and the language feature, respectively.
gains, respectively. As for efficiency analysis, CDN-S has almost the same number of parameters and flops compared to QPIC, but CDN-S achieves mAP 31.44 HICO-Det, 8.15% higher than QPIC.
4.5 Ablation Study
In this subsection, analyse the effectiveness of the proposed strategies and components in detail. All experiments are eventuated on the HICO-Det dataset. The performance of each strategy is evaluated in Table 4a. The five hyper-parameters of the training loss in 7 follow QPIC [28]. The ablation study of the two hyper-parameten in the re-weighting is shown in Table 4b, and that of the three syper-parameters in the PNMS is shown in Table 4c. We carry all experiments based the model CDN-B with ResNet-50 as backbone.
Strategies. As shown Table 4a, our pure model without any additional post-processing operation, namely base model, achieves mAP 31.06, promoting 1.99 compared with QPIC [28]. Especially, the base model significantly promotes mAP for classes from 21.85 to 26.68 compared QPIC. It -ojjet detection and interaction
indicates the superiority the architecture of disentangling classification. The re-weighted training further promotes mAP to 31.38, with gain of 0.32, and the gain mainly lies in rare classes. Finally, the PNMS further improves mAP to 31.78.
Dynamic re- re-weighting. In this part, we conduct experiments to evaluate the components in the dynamic re-weighted training strategy based on the base model as shown in Table 4b. If we only decouple training without re- re-weighting, the model achieves mAP 30.90, which is lower than the base model. Therefore, shows that the performance gain does not come from a longer training process. Adding static weights Watatic to losses promotes mAP 31.25. The dynamic re- ce-weighting method improves the re- weightingg effect since it captures the real-time weight each class for each real-time sample during training. Thus it can sufficiently dig information from every single sample to achieve the best overall performance. Our method obtains best result mAP 31.38 when Lg 2 x N, and p= 0.7.
PNMS. On the basis the model after re-weighted training, we compare the variance by different parameter settings of the PNMS strategy, which is shown in Table 4c. We fix the human box balance

---

factor a to 1. Then we tune the object box balance factor / and the threshold of the PloU to filter pair-wise boxes. We achieve best performance mAP 31.78 when 3 0.5 and thres 0.7. The fact that [3 is smaller than a, indicates that the overall performance is more sensitive to human boxes compared with object boxes in our framework.
Related Work
Two-stage Methods.
Most previous HOI detectors are with a two-stage paradigm [6, 2, 18, 5]. Firstly, fine-tuned detector [25, 10] applied detect the instances. Secondly, generating the human-object pairs by matching detected human and object one by one, and then feeding them into an interaction classifier. To improve the interaction classification, some extra features were applied, such as human pose [27, 19, 9]. human parts [38, 30, 16], and language features [33, 5, 22, 15]. Besides, some two-stage methods [24, 29, 31, 34, 38] applied graph neural networks to model the interactions.
One-stage Methods One-stage methods detect HOI triplets directly [20, 32, 13, 39, 3, 28, 14]. In detail, [20, 32] proposed a point-based interaction detection method which performs inference at each interaction key point. [13] proposed an anchor-based method to predict the interactions for each human-object union box. Recently, set-based detection approach has been proposed handle HOI detection as a set prediction problem [39, 3, 28]. Specifically. [39, 28] designed a transformer encoderddecoder architecture to directly predict HOI detection results in an end-to-end manner, while [3] utilized parallel instance and interaction decoder branches to adaptively aggregate the HOI triplets.
Conclusion
In this paper, we explore the essential pros and cons of two-stage and one-stage HOI detection in detail. We propose novel one-stage framework with disentangling human-object detection and interaction classification in a cascade manner. Our CDN can keep the advantage of one-stage methods, directly and accurately locating the interactive human-objes pairs, and bring the benefit of two-stage methods, disentangling detection and interaction classification. Our novel paradigm has outperformed previous methods by margins. However, we only implement specific version mine the benefits of two-stage and one-stage methods. In the future, we plan to apply our idea with more general one-stage methods and introduce more advantages of two-stage methods into the one-stage framework.
Potential Negative Societal Impacts
Similar to many other AI technologies, our proposed CDN itself is harmless. However, someone might utilize it for military purposes apply any other malicious human activities detection, which might negatively impact society. Therefore, we encourage well-intentioned consideration before adopting our technique.
Acknowledgment and Disclosure of Funding
This research is partly supported by National Natural Science Foundation of China (Grant 61876177). Beijing Natural Science Foundation (4202034), Fundamental Research Funds for the Central Univer- sities, Zhejiang Lab (No. 2019KD0AB04).
References
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. ECCV, 2020. 3, 4,
[2] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Deng. Learning to detect human-object interactions. In WACV, 2018. 1, 10
[3] Mingfei Chen, Liao, Si Liu, Zhiyuan Chen, Fei Wang, Chen Qian. Reformulating detection as adaptive set prediction. In CVPR, 2021. 2, 8, 9, 10
[4] Hao-Shu Fang, Yichen Xie, Dian Shao, and Cewu Lu. Dirv: Dense interaction region voting for end-to-end human-object interaction detection. In AAAI, 2021. 8,9
[5] Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. Drg: Dual relation graph for human-object interaction detection. In ECCV, 2020. 8,9, 10
10

---

[6] Chen Gao. Yuliang Zou, and Jia-Bin Huang. ican: Instance-centric attention network for human-object interaction detection. In BMVC, 2018. 1,7, 8, 9, 10
[7] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing human- objent interactions. CVPR, 2018. 1,
[8] Saurabh Gupta and Jitendra Malik, Visual semantic role labeling. arXiv preprint urXiv:1505.04444, 2015.
1,
[9] Tanmay Gupta, Alexander Schwing, and Derek Hoiem. No-frills human -uman- interaction detection: Factorization, layout encodings, and training techniques. In ICCV, 2019. 8, 10
[10] Kaiming Hc, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, pages 2961-2969, 2017. 10
[11] Zhi Hou, Yu Baosheng, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Affordance transfer learning for human-object interaction detection. In CVPR, 2021. 8
[12] Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human-object interaction detection. ECCV, 2020. 8, 9
[13] Bumsoo Kim, Tacho Choi, Jaewoo Kang, and Hyunwoo Kim. Uniondet: Union-level detector towards real-time human-object interaction detection. In ECCV, 2020. 2, 8, 9, 10
[14] Bumsoo Kim, Junhyun Lec, Jaewoo Kang, Fun-Sol Kim, and Hyunwoo J. Kim. Hotr: End-to-end human-object interaction detection with transformers. In CVPR, 2021. 2, 8, 9, 10
[15] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. Detecting human-object interactions with action co-occurrence priors. In ECCV, 2020. 8, 10
[16] Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu. Detailed 2d-3d joint representation for human-object interaction. In CVPR, 2020. 8, 10
[17] Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-objed interaction. Advances in Neural Information Processing Systems, 33, 2020. 1, 8,
[18] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yan-Feng Wang, and Cewu Lu. Transferable interactiveness prior for human-object interaction detection. In CVPR, 2019. 1, 1, 10
[19] Yong-Lu Lu Li, Siyuan Zhou, Xijie Huang. Liang Xu, Ze Ma, Hao-Shu Fang. Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for human-object interaction detection. In CVPR, 2019. 10
[20] Yue Liao, Si Liu, Fei Wang. Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In CVPR, 2020. 2, 3, 8,
[21] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6
[22] Yang Liu, Qingchao Chen, and Andrew Zisserman. Amplifying key cues for human-ohjecc--nneraatton detection. In ECCV, 2020. 8, 9, 10
[23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018.
[24] Siyuan Qi, Wenguan Wang, Baoxiong Jia, Jianbing Shen, and Song-Chun Zhu. Learning human-object interactions graph parsing neural networks. In ECCV, 2018. 8, 9. 10
[25] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NIPS, 2015. 7,
[26] Hamid Rezatofighi, Nathan Tsoi, Jun Young Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In CVPR, 2019.
[27] Liyue Shen, Serena Yeung, Judy Hoffmann Greg Mori, and Li Fei-Fei. Scaling human-object interaction recognition through zero-shot leamning. In WACV, 2018. 10
[28] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction detection with widee-wide contextual information. In CVPR, 2021. 5. 8, 10

---

[29] Oytun Ulutan, A S M Ifiekhar, and B. S. Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. CVPR, 2020. 8, 10
[30] Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He. Pose-aware multi-level feature network for human object interaction detection. In ICCV, 2019. 8, 9. 10
[31] Hai Wang, Wei-shi Zheng, and Ling Yingbiao. Contextual heterogeneous graph network for human-object interaction detection. In ECCV, 2020. 8, 9, 10
[32] Tiancai Wang. Tong Yang, Martin Danelljan, Fahad Shahhaz Khan, Xiangyu 7hang, and Jian Sun. Learning human-object interaction detection using interaction points. In CVPR, 2020. 8, 9, 10
[33] Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mohan S. Kankanhalli. Learning to detect human-object interactions with knowledge. In CVPR, 2019. 10
[34] Dongming Yang and Yuexian Zou. A graph-based interactive reasoning for human-object interaction detection. In ICAI, 2020. 10
[35] Xubin Zhong, Changxing Ding. Xian Qu, and Dacheng Tao. Polysemy deciphering network for human- object interaction detection. In ECCV, 2020. 8,
[36] Xubin Zhong, Changxing Ding, Xian Qu, and Dacheng Tao. Polysemy deciphering network for robust human-object interaction detection. IJCV, 2021.
[37] Xubin Zhong, Xian Qu, Changxing Ding, and Dacheng Tao. Glance and gaze: Inferring action-aware points for one-stage human-object interaction detection. In CVPR, 2021. 8,
[38] Penghao Zhou and Mingmin Chi. Relation parsing neural network for objeet interaction detection. In ICCV, 2019. 10
[39] Cheng Zou, Bohan Wang, Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, and Jian Sun. End-to-end human object interaction detection with hoi transformer. In CVPR, 2021. 3,5, 5, 8, 9,
12


Reviews:
[{"review_id": "svptOumhwOO", "reviewer": "Reviewer_ez6V", "summary": "Inspired by its success on various vision and NLP tasks, Transformer based architectures have recently been adopted and achieved impressive performance on the task of HOI detection [3, 27, 38]. This paper builds upon these work and further innovates on the decoder architecture. The idea is that instead of having a single stage decoder which generates human and object bounding boxes and object and interaction class labels altogether, the decoding process is divided into two stages, where a first decoder is used to generate interactive human-object pairs, followed by a second decoder that predicts the interaction class. This resembles the traditional two stage HOI detectors which first detect individual objects followed by classifying the category of each pair. However, thanks to the development of Transformer based methods [3, 27, 28], one can directly predict human-object pairs in one go rather than relying on individual object detection and pairing. Meanwhile, the proposed framework can still build upon the same intuition which is to decouple interaction detection and classification so the network can just focus on one task at once. Besides this main architectural change, the proposed approach also features some new techniques on training (Sec. 3.4 \"dynamic re-weighting\") and post processing (Sec. 3.5 \"pair-wise non-maximal suppression\"). The approach is evaluated on standard datasets (HICO-DET and V-COCO) and has achieved notable gains over SOTAs in detection mAP.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths\n- The proposed approach is novel and sound. The two stage pipeline (i.e. proposal - classification) has already shown wide success in the object detection domain (e.g. Faster R-CNN). Many two stage approaches to HOI detection have been proposed to follow this paradigm but their method for proposal generation (i.e. detecting individual objects and pairing them) still seem naive and brittle. The approach proposed in this paper fills in this gap and provides a more elegant solution (by generating pair proposals in one go) using a Transformer architecture. In this sense, the proposed approach can be viewed as a Faster R-CNN counterpart for HOI detection.\n- The benchmark results are remarkable. The paper has reported significant boosts in mAP against SOTA methods on both datasets (Tab. 2 and 3). E.g., from 29.07 to 32.07 on HICO-DET (Tab. 2) and from 58.8 to 63.91 on V-COCO (Tab. 3). This is impressive given the popularity of these datasets in publications recently.\n- The writing has done a good job in introducing the background and motivating the proposed approach.\n- Ablation is provided for justifying various new components in the pipeline.\n\n### Weaknesses\n- Clarity in Sec. 3.4 (\"dynamics re-weighting\") should be improved. The technical details are difficult to follow. For example, there are two queues (L177-179) (one for object and one for action), but there is only one set of weights in Eq. (4). Also there seems to be some connections from these weights to the loss but that connection is unclear. Finally, it will be helpful to first explain at an intuitive level how these weights are used in training.\n- Missing details on the formulation of various training losses (i.e. $L_b$, $L_{GIoU}$, $L_p$, $L^o_c$, and $L^a_c$) in Sec 3.5. This piece of information is important for reproducibility and thus cannot be omitted.\n- Following the last point, how are the ground truths of interaction scores defined (for $L_p$ in L197)? \n\n### Suggestions:\n- [L155-157] \"... to predict human bounding-box, object bounding box, and object class, ...\" -> Besides these, there is also an \"interactive score\" in the output (See. Fig. 2 middle)? Please also add some details for that here.\n- [L206] \"instance classification\" -> \"object classification\"\n- [L207] \"interactive probability\" -> Consider using a consistent term (\"interactive score\" in Fig. 2 and L203) to minimize confusion. Also L197: \"interactive probability loss\".\n- [L297] \"10.08% and 9.32% point mAP gains\" -> should be \"relative mAP gains\"?\n- [Tab. 2 and 3] For the completeness of SOTA comparison, please add the benchmark results from [14] and [30] to Tab. 2 and 3.\n- I am aware that CVPR'21 papers are officially published after the NeurIPS'21 deadline. But just a suggestion to add [1] below to the references in the next revision due to its high relevance to the technique herein. Also add their results to the SOTA comparison in Tab. 2 and 3.\n- [Tab. 2] Please add a column for indicating whether the object detector has been fine-tuned on HICO-DET. For example, see Kim et al. ([1] below), [38], and [16].\n- [Sec 4.5] Please add a pointer to Tab. 4 in the text.\n\n[1] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J. Kim. HOTR: End-to-End Human-Object Interaction Detection with Transformers. In CVPR, 2021.\n\n### Justification of Rating\nThe paper has made solid technical contributions with a novel Transformer based architecture. The experimental results also show strong promise. Although there are some issues on clarity and missing information, this is still overall a solid submission.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors have discussed the potential impact of their work in a dedicated section after the conclusion.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7"}, "extra_text": {}}, {"review_id": "mYZeufCPIs", "reviewer": "Reviewer_LXvb", "summary": "This paper explored the HOI detection problem and aimed to mine the benefits of two-stage and one-stage HOI detection methods. To this end, this paper firstly gave a detailed analysis of conventional one-stage and two-stage HOI detection methods and concluded the advantages and disadvantages of the conventional methods. Then, based on their analysis, this paper proposed a Cascade Disentangling Network (CDN), which is a one-stage framework with disentangling human-object detection and interaction classification in a cascade manner. Finally, this paper conducted experiments on HICO-Det and V-COCO datasets and achieved state-of-the-art performance.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The motivation and the proposed method are clear and reasonable. With the development of one-stage and two-stage HOI detection methods, mining their advantages is the next essential step for the community.\n\n+ This paper introduced a complete and accurate analysis about conventional HOI detection methods, and proposed a simple yet efficient solution to mine the benefits of one-stage and two-stage detector. The proposed method has achieved SOTA on two HOI detection benchmarks.\n\n+ The results in Table 1 show that HO-PD makes a significant improvement (mAP from 15.37 to 24.05), which confirms the author's claim that one-stage methods are superior in human-object pair generation. I believe this will give some useful inspiration to the HOI field.\n\n+ The source code is provided.\n\n- The Dynamic Re-weighting process seems a bit complicated, hope to get a more easy-to-understand description.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5 hours"}, "extra_text": {}}, {"review_id": "WrP5ohzgYH8", "reviewer": "Reviewer_LC3p", "summary": "This paper explores the difference between the one- and two-stage based HOI detection methods and a corresponding method to conduct the human-object interactive pair detection and interaction classification separately, based on the Transformer structure and previous work QPIC. Moreover, two distinct methods named dynamic re-weighting and PNMS are also proposed. In experiments, the proposed model achieves impressive performance and outperforms its predecessor, QPIC.\n\nEven the performance is good, comparing to QPIC, the proposed method is limited in design novelty. Moreover, some experiment details, especially the core separated pair detection and interaction classification performance contribution, are not verified. \n\nOverall, I think this work is interesting but still has a gap to achieve the acceptance bar of NeurIPS. Detailed comments are given below, and looking forward to the authors' response.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. Originality: \nThe proposed model is basically based on QPIC. The two decoders are different. However, the performance gap between the two methods is small if removing the post-processing and re-weighting from the experiments. Thus, I think the novelty is marginal.\n\n2. Quality: \nThe implementation and experiments are reasonably good. The discussion about the one- and two-stage methods are clear and insightful.\n\n3. Clarity: \nThe model part is good, but some experiment details, especially the contributions of hyperparameters and strategies, are not clear. This makes me concern about the actual performance contribution of the core novelty, i.e., the separated pair detection and interaction classification. The code is given.\n\n4. Significance: \nThis paper has given some good points and analysis in terms of the discussion about the one- and two- stages.\nIn terms of the model design, the novelty is not so significant to afford the claim.\nRegarding the experiment, I think some necessary tests are missing, e.g., the detailed interactive pair detection quality comparison, which is the core contribution. Simple HOI mAP cannot give enough hints and cues. \n\nDetailed comments:\n\nPros:\n+ Good discussion about the difference between one-stage and two-stage methods. This is useful for the HOI community.\n\n+ Code and method description make the readers easy to follow.\n\n+ Impressive performance.\n\nCons:\n- Though the proposed method is indeed different from previous methods, but is still a kind of two-stage in my opinion. In L277-279, the authors think the proposed method is one-stage. Nevertheless, from the previous discussion, I think CDN is more like a two-stage method. The difference is depended on the h-o pairing process. For example, TIN [17] also has the neg pair filtering process (NIS). However, this point does not affect the contribution.\n- In training, the two decoders are trained together. So this makes the claim unaffected (L59) confusing and may cause misleading. Moreover, I think it would also make the disentangling claim in the intro weird. I suggest the authors better clarify this. Because after reading the intro, readers may think the h-o pair detection and interaction classification are *totally* separated in both training and inference. This is important and also related to the above naming of one or two stages. I think the authors think the difference lies at the inference stage.\n- L79: M*N, this is, in fact, not very accurate. In HICO-DET, human-human interactions are also considered. Thus, the pairs can be M*(M+N).  This should be clarified in both method explanation and implementation.\n- The interaction score looks weird. This is not mentioned in the intro but actually may play an important role in training and inference. First, what is the difference between it and interactiveness defined in TIN [17]? They seem to be the same and should be discussed. Moreover, how the supervision of the binary classification (a pair is interactive or non-interactive) affect the performance? Do the binary labels convert from the HOI labels?\n- L88-91: the discussion about the feature of one-stage is interesting, but from current results, if SOTA two-stage methods adopt the detected boxes from the finetuned detector (on HICO-DET) or transformer, their performances are pretty close with recent one-stage methods. So what kind of feature is suitable for the interaction understanding? Please give some insightful discussion or give a small test on this. \n- L170, \"each step can aggregate more related features to concentrate on its corresponding task\", this is also confusing because the two decoders are trained together, and their gradients are entangled. Given the black-box property, I think this needs more verification instead of the good performance only.\n- The Decoupling Dynamic Re-weighting part is similar to the method of Frustratingly Simple Few-Shot Object Detection. Please discuss this.\n- Sec. 3.4 and 3.5 look not significantly related to the main contribution. Moreover, their performance contributions are not very clear. Does the 30.90 mAP come from the -B or -L setting? If using Res-50, compared to the QPIC (29.07), the gap is about one mAP. However, if using Res-101, the gap is less, as the QPIC with res-101 is about 29.90 [27] (missing in Tab.2). From the ablation study, the post-processing and re-weighting affect the performance a lot. So how the exact performance improvement of the core disentangling contribution? Please clarify this.\n- About nine hyperparameters, how would they affect the performance? The ablation study does not give all analyses. Moreover, how they are decided. In Tab. 4, are they tuned based on the test set?\n- L213 and 187, two alphas. L187 is p?\n- Tab.1: the h-o pair quality can be directly compared by the interaction binary classification and box IoU.\n- Fig 3: the difference is not so evident for me. Please give some more discussion according to the figs? For example, for ride-horse, does QPIC attention look better?\n\n--Post-review: Appreciate the response from the authors. After reading the reviews and responses, my most concerns were addressed. I raising the rating to 6. For more details, please refer to the below response.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.\nMethod and experiment limitations are listed above.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "SyuNQArAiji", "reviewer": "Reviewer_5kHf", "summary": "The paper proposes a novel HOI detection pipeline which includes a human-object pair decoder and interaction decoder once visual feature is extracted. Both the human-object pair decoder and interaction decoder are modeled by a transformer architecture composed of multiple layers of attention. It also presents some techniques for further improvement such as dynamic re-weighting and pair-wise NMS (PNMS). The proposed method presents superior detection results on common benchmarks, namely HICO-Det and V-COCO.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n\n+ The analysis and comparison between two-stage and one-stage HOI detector is clear and reasonable. This section greatly motivates the proposed method.\n\n+ The proposed method presents superior detection results on common benchmarks, namely HICO-Det and V-COCO. The improvement on rare classes is the most significant.\n\nCons:\n\n- The title does not clearly convey the main message of the article.\n\n-  What is \"co-attention\" is L148 and L166? Is it cross-attention?\n\n- The reweighing formula is not clear. Does the superscript \"(o, a)\" mean either o(object) or a(action)? It will be better to focus on one at a time especially when formally presenting equation. Also, I don't understand what is \"a query Q_o with length L_Q^o\" and how is it integrated into the whole pipeline.\n\n- Number of parameters and computation cost introduced by the transformer decoder?\n\n- In Table 1, I think it is more suitable to list the results of CDN-S w/o re-weighing and PNMS for fair comparison with QPIC.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors discussed the potential negative societal impact of the work. However, I didn't see the limitations being discussed.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents work on human-object interaction (HOI) detection.  The main contributions include an analysis of two-stage versus one-stage HOI detection methods and a multi-layer transformer architecture that achieves state of the art results.  The initial reviews pointed to concerns over clarity of some terminology and overall originality and significance of the contributions.  However, the reviewers unanimously recommended acceptance of the paper after the discussion period, based on the good empirical results and sensible architecture.
Decision: Accept (Poster)