Mining the Benefits of Two-stage and  One-stage HOI Detection

1 Introduction . The goal of Human-Object Interaction ( HOI ) detection [ 2 , 20 , 6 , 18 , 7 , 8 , 17 , 3 ] is to make a machine detailedly understand human activities from a static image . Human activities in this task are abstracted as a set of < human , object , action > HOI triplets . Thus , an HOI detector is required to locate humanobject pairs and classify their corresponding action simultaneously . Based on this definition , we can summarize conventional HOI detection methods into two paradigms , i.e. , two-stage methods , and one-stage methods . These two paradigms have made significant progress with the development of deep learning , but both paradigms still have their shortcomings due to their structural design . This paper aims to present a detailed analysis of methods under these two paradigms and propose a solution to mine the benefits of two-stage and one-stage methods . We first take a closer look at the conventional two-stage and one-stage HOI detectors . Conventional two-stage methods [ 6 , 2 , 18 , 5 ] are mostly with a serial architecture . As shown in Figure 1 ( a ) , two-stage methods detect humans and objects first and then feeds the human-object pairs , which are generated by matching humans and objects one by one , into an interaction classifier . The serial architecture suffers from locating the interactive human-object pairs under the interference of a large number of negative pairs only based on local region features . Otherwise , the efficiency of ∗Equal contribution †Corresponding author ( liusi @ buaa.edu.cn ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . two-stage methods is also limited by the serial architecture . To alleviate these problems , one-stage methods [ 20 , 13 , 39 , 3 , 28 , 14 ] are proposed to detect the HOI triplets directly and break HOI detection as multi-task learning , i.e. , human-object detection and interaction classification , which is shown in Figure 1 ( b ) . Therefore , one-stage methods can easily focus on the interactive human-object pairs and effectively extract corresponding features in an end-to-end manner . However , it is difficult for a single model to make a good trade-off on multi-task learning since human-object detection and interaction classification are two very different tasks , which requires the model to focus on different visual features . As shown in Figure 1 ( c ) , though some previous methods [ 20 , 3 ] design two parallel branches to detect instances and predict interaction respectively , the interaction classification branch still needs to regress additional offsets to associate humans and objects . Thus the interaction branch is also required to make a trade-off between interaction classification and human and object positioning . Therefore , the intuitive idea is to take the essence and discard the dregs from the two paradigms . To attain this , we propose a novel end-to-end one-stage framework with disentangling human-object detection and interaction classification in a cascade manner , namely Cascade Disentangling Network ( CDN ) . The original intention of our framework is to keep the advantages of conventional onestage methods , directly and accurately locating the interactive human-object pairs , and bring the advantages of two-stage methods into our one-stage framework , disentangling human-object detection and interaction classification . As shown in Figure 1 ( d ) , in our proposed framework , we design a human-object pair decoder based on the one-stage paradigm by removing the interaction classification function , namely HO-PD , and following an isolated interaction classifier . To instantiate our idea with an end-to-end manner , we design the HO-PD based on the previous state-of-the-art one-stage transformer-based HOI detector , HOI-Trans [ 39 ] and QPIC [ 28 ] , where we remove the interaction classification head for each query and make it focus on human-object pairs detection . Otherwise , we design an independent HOI decoder for interaction classification to make the interaction classification unaffected by human-object detection . Therefore , there exists a core problem , i.e. , how to link the human-object pair and the corresponding action class . To address this problem , we initialize the query embedding of the HOI decoder with the output of the last layer of the HO-PD . In this case , the HOI decoder is able to learn the corresponding action category under the guidance of the query embedding and free out from the human-object detection task . Moreover , we design a decoupling dynamic re-weighting manner to handle the long-tailed problems in HOI detection . Our contributions can be summarized threefold : ( 1 ) We conduct a detailed analysis of two conventional HOI detection paradigms , i.e. , two-stage and one-stage . ( 2 ) We propose a novel one-stage framework with a cascade disentangling decoder to combine the advantages of two-stage and onestage methods . ( 3 ) Our method outperforms previous state-of-the-art methods by a large margin on the HOI detection task , especially achieves a 25.35 % performance gain on rare classes of HICO-Det . 2 Analysis of Two-stage and One-stage HOI detectors . We first introduce a unified formulation for the HOI detection problem . Given a human-centric image I , the model T ( · ) is required to predict a set of HOI triplets S = { ( bhi , boi , ai ) , i ∈ { 1 , 2 , · · · , K } } , where bhi , b o i and ai denotes a human bounding-box , an object bounding-box and their corresponding action category , respectively . Two-stage HOI detector . Two-stage detectors can be regarded as an instance-driven manner , detecting instances first and predicting interaction based on the detected instances . The two-stage detector divides T ( · ) into two stages , i.e. , detection Td ( · ) and interaction classification Tc ( · ) . In the first stage , we suppose that Td ( · ) produces M human bounding-boxes and N object bounding-boxes . Here the ‘ object ’ is a universal object which includes human as one class . Therefore , Td ( · ) generates M ×N human-object pairs . In general , the number of true-positive interactive human-object pairs , denoted as K ′ , is much smaller than M ×N . However , in the second stage , Tc ( · ) needs to scan all M ×N pairs one by one and predict an action category with its corresponding confidence score . In this case , Tc ( · ) is required to inference M ×N times to find K ′ interactive pairs from M ×N pairs . We argue that this manner causes three problems . Firstly , these models produce a more additional computational cost , whose time complexity is O ( M ×N ) O ( K ′ ) . Secondly , the imbalance between positive and negative samples makes the model easily overfit to negative samples . Thus the model tends to assign a ‘ no-interaction ’ class for human-object pairs with very high confidence , suppressing the true-positive samples . Thirdly , the accuracy of interaction classification is influenced by the non-end-to-end pipeline . Because the interaction classification is mostly based on the region features extracted by Td ( · ) , while the core of Td ( · ) is to regress bounding-boxes and its extracted features pay more attention to the edge of regions , thereby such features are not good options for interaction classification , which needs more context . However , it is an excellent property for two-stage methods that disentangling detection and interaction classification makes each stage focus on its task and produce good results in each stage . One-stage HOI detector . As for one-stage methods , they detect all HOI triplets S directly and simultaneously with an end-to-end framework . Such paradigm has greatly relieved the three problems of two-stage methods , especially for efficiency , where the time complexity is reduced to O ( K ′ ) . Most one-stage methods are interaction-driven , which directly locate the interaction point [ 20 ] or interactive human-object pairs [ 39 ] , thereby reducing negative sample interference . However , coupling human-object detection and interaction classification limit their performance because it is hard to generate a unified feature representation for two very different tasks . Though the parallel one-stage methods break HOI detection into two parallel branches , their interaction branch still suffers from multi-task learning . Specifically , the optimization target of interaction branch is P ( eh , eo , a|V ) , where eh and eo are associative embeddings , e.g. , offset , to match interaction with human and object respectively . Therefore , even if detection is organized as an independent branch , the interaction branch must position humans and objects for the association . The set-based detectors couple detection and interaction completely , whose optimization function is P ( bh , bo , a|V ) . Next , we introduce a simple one-stage framework with disentangling human-object detection and interaction classification , namely CDN , to mine the benefits of two-stage and one-stage HOI detectors . Our CDN disentangles the original set-based one-stage optimization function into two cascade decoders . Firstly , we predict human-object pair by P ( bh , bo|V ) . Secondly , we apply an isolated decoder to predict the action category by P ( a|V , bh , bo ) . More details are in the following . 3 Method . In this section , we will present a detailed introduction to the pipeline of our proposed CDN . In section 3.1 , we present an overview of our framework and briefly introduce the pipeline . In section 3.2 , we introduce the visual feature extractor . The cascade disentangling HOI decoder is introduced in section 3.3 . Section 3.4 introduces a novel dynamic re-weighting mechanism that mitigates the long-tailed problem . The detailed training process and post-processing are discussed in section 3.5 . 3.1 Overview . The architecture of our proposed CDN is illustrated in Figure 2 . Our CDN is organized in a cascade manner with a visual feature extractor . Given an image , we first follow transformer-based detection methods [ 1 , 39 ] to apply a CNN followed by a transformer encoder architecture to extract visual features into a sequence . Then we detect HOI triplets in two cascade decoders . Firstly , we apply the Human-Object Pair Decoder ( HO-PD ) to predict a set of human-object bounding-boxes pairs based on a set of learnable queries . Next , taking the output of the last layer of HO-PD as queries , an isolated interaction decoder is utilized to predict the action category for each query . Finally , the HOI triplets are formed by the output of the above two cascade decoders . 3.2 Visual Feature Extractor . We define the visual feature extractor by combining a CNN and a transformer encoder . Fed with an input image I with shape ( H , W , C ) , the CNN generates a feature map of shape ( H ′ , W ′ , Db ) . Then , Db is reduced to Dc by a projection convolution layer with a kernel size 1× 1 . Next , a flatten operator is used to generating the flatten feature Xv ∈ R ( H ′ ×W ′ ) ×Dc by collapsing the spatial dimensions into one dimension . This flatten feature is then fed into a transformer encoder and the position encoding Epos ∈ R ( H ′ ×W ′ ) ×Dc , which distinguishes the relative position in the sequence Xs ∈ R ( H ′ ×W ′ ) ×Dc . Thanks to the multi-head self-attention mechanism , the transformer encoder produces a feature map with richer contextual information by summarizing global information . The output of the encoder is denoted as global memory with a dimension of Dc .

Summary: Inspired by its success on various vision and NLP tasks, Transformer based architectures have recently been adopted and achieved impressive performance on the task of HOI detection [3, 27, 38]. This paper builds upon these work and further innovates on the decoder architecture. The idea is that instead of having a single stage decoder which generates human and object bounding boxes and object and interaction class labels altogether, the decoding process is divided into two stages, where a first decoder is used to generate interactive human-object pairs, followed by a second decoder that predicts the interaction class. This resembles the traditional two stage HOI detectors which first detect individual objects followed by classifying the category of each pair. However, thanks to the development of Transformer based methods [3, 27, 28], one can directly predict human-object pairs in one go rather than relying on individual object detection and pairing. Meanwhile, the proposed framework can still build upon the same intuition which is to decouple interaction detection and classification so the network can just focus on one task at once. Besides this main architectural change, the proposed approach also features some new techniques on training (Sec. 3.4 "dynamic re-weighting") and post processing (Sec. 3.5 "pair-wise non-maximal suppression"). The approach is evaluated on standard datasets (HICO-DET and V-COCO) and has achieved notable gains over SOTAs in detection mAP.