Mining the Benefits of Two-stage and  One-stage HOI Detection

1 Introduction . The goal of Human-Object Interaction ( HOI ) detection [ 2 , 20 , 6 , 18 , 7 , 8 , 17 , 3 ] is to make a machine detailedly understand human activities from a static image . Human activities in this task are abstracted as a set of < human , object , action > HOI triplets . Thus , an HOI detector is required to locate humanobject pairs and classify their corresponding action simultaneously . Based on this definition , we can summarize conventional HOI detection methods into two paradigms , i.e. , two-stage methods , and one-stage methods . These two paradigms have made significant progress with the development of deep learning , but both paradigms still have their shortcomings due to their structural design . This paper aims to present a detailed analysis of methods under these two paradigms and propose a solution to mine the benefits of two-stage and one-stage methods . We first take a closer look at the conventional two-stage and one-stage HOI detectors . Conventional two-stage methods [ 6 , 2 , 18 , 5 ] are mostly with a serial architecture . As shown in Figure 1 ( a ) , two-stage methods detect humans and objects first and then feeds the human-object pairs , which are generated by matching humans and objects one by one , into an interaction classifier . The serial architecture suffers from locating the interactive human-object pairs under the interference of a large number of negative pairs only based on local region features . Otherwise , the efficiency of ∗Equal contribution †Corresponding author ( liusi @ buaa.edu.cn ) 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . two-stage methods is also limited by the serial architecture . To alleviate these problems , one-stage methods [ 20 , 13 , 39 , 3 , 28 , 14 ] are proposed to detect the HOI triplets directly and break HOI detection as multi-task learning , i.e. , human-object detection and interaction classification , which is shown in Figure 1 ( b ) . Therefore , one-stage methods can easily focus on the interactive human-object pairs and effectively extract corresponding features in an end-to-end manner . However , it is difficult for a single model to make a good trade-off on multi-task learning since human-object detection and interaction classification are two very different tasks , which requires the model to focus on different visual features . As shown in Figure 1 ( c ) , though some previous methods [ 20 , 3 ] design two parallel branches to detect instances and predict interaction respectively , the interaction classification branch still needs to regress additional offsets to associate humans and objects . Thus the interaction branch is also required to make a trade-off between interaction classification and human and object positioning . Therefore , the intuitive idea is to take the essence and discard the dregs from the two paradigms . To attain this , we propose a novel end-to-end one-stage framework with disentangling human-object detection and interaction classification in a cascade manner , namely Cascade Disentangling Network ( CDN ) . The original intention of our framework is to keep the advantages of conventional onestage methods , directly and accurately locating the interactive human-object pairs , and bring the advantages of two-stage methods into our one-stage framework , disentangling human-object detection and interaction classification . As shown in Figure 1 ( d ) , in our proposed framework , we design a human-object pair decoder based on the one-stage paradigm by removing the interaction classification function , namely HO-PD , and following an isolated interaction classifier . To instantiate our idea with an end-to-end manner , we design the HO-PD based on the previous state-of-the-art one-stage transformer-based HOI detector , HOI-Trans [ 39 ] and QPIC [ 28 ] , where we remove the interaction classification head for each query and make it focus on human-object pairs detection . Otherwise , we design an independent HOI decoder for interaction classification to make the interaction classification unaffected by human-object detection . Therefore , there exists a core problem , i.e. , how to link the human-object pair and the corresponding action class . To address this problem , we initialize the query embedding of the HOI decoder with the output of the last layer of the HO-PD . In this case , the HOI decoder is able to learn the corresponding action category under the guidance of the query embedding and free out from the human-object detection task . Moreover , we design a decoupling dynamic re-weighting manner to handle the long-tailed problems in HOI detection . Our contributions can be summarized threefold : ( 1 ) We conduct a detailed analysis of two conventional HOI detection paradigms , i.e. , two-stage and one-stage . ( 2 ) We propose a novel one-stage framework with a cascade disentangling decoder to combine the advantages of two-stage and onestage methods . ( 3 ) Our method outperforms previous state-of-the-art methods by a large margin on the HOI detection task , especially achieves a 25.35 % performance gain on rare classes of HICO-Det . 2 Analysis of Two-stage and One-stage HOI detectors . We first introduce a unified formulation for the HOI detection problem . Given a human-centric image I , the model T ( · ) is required to predict a set of HOI triplets S = { ( bhi , boi , ai ) , i ∈ { 1 , 2 , · · · , K } } , where bhi , b o i and ai denotes a human bounding-box , an object bounding-box and their corresponding action category , respectively . Two-stage HOI detector . Two-stage detectors can be regarded as an instance-driven manner , detecting instances first and predicting interaction based on the detected instances . The two-stage detector divides T ( · ) into two stages , i.e. , detection Td ( · ) and interaction classification Tc ( · ) . In the first stage , we suppose that Td ( · ) produces M human bounding-boxes and N object bounding-boxes . Here the ‘ object ’ is a universal object which includes human as one class . Therefore , Td ( · ) generates M ×N human-object pairs . In general , the number of true-positive interactive human-object pairs , denoted as K ′ , is much smaller than M ×N . However , in the second stage , Tc ( · ) needs to scan all M ×N pairs one by one and predict an action category with its corresponding confidence score . In this case , Tc ( · ) is required to inference M ×N times to find K ′ interactive pairs from M ×N pairs . We argue that this manner causes three problems . Firstly , these models produce a more additional computational cost , whose time complexity is O ( M ×N ) O ( K ′ ) . Secondly , the imbalance between positive and negative samples makes the model easily overfit to negative samples . Thus the model tends to assign a ‘ no-interaction ’ class for human-object pairs with very high confidence , suppressing the true-positive samples . Thirdly , the accuracy of interaction classification is influenced by the non-end-to-end pipeline . Because the interaction classification is mostly based on the region features extracted by Td ( · ) , while the core of Td ( · ) is to regress bounding-boxes and its extracted features pay more attention to the edge of regions , thereby such features are not good options for interaction classification , which needs more context . However , it is an excellent property for two-stage methods that disentangling detection and interaction classification makes each stage focus on its task and produce good results in each stage . One-stage HOI detector . As for one-stage methods , they detect all HOI triplets S directly and simultaneously with an end-to-end framework . Such paradigm has greatly relieved the three problems of two-stage methods , especially for efficiency , where the time complexity is reduced to O ( K ′ ) . Most one-stage methods are interaction-driven , which directly locate the interaction point [ 20 ] or interactive human-object pairs [ 39 ] , thereby reducing negative sample interference . However , coupling human-object detection and interaction classification limit their performance because it is hard to generate a unified feature representation for two very different tasks . Though the parallel one-stage methods break HOI detection into two parallel branches , their interaction branch still suffers from multi-task learning . Specifically , the optimization target of interaction branch is P ( eh , eo , a|V ) , where eh and eo are associative embeddings , e.g. , offset , to match interaction with human and object respectively . Therefore , even if detection is organized as an independent branch , the interaction branch must position humans and objects for the association . The set-based detectors couple detection and interaction completely , whose optimization function is P ( bh , bo , a|V ) . Next , we introduce a simple one-stage framework with disentangling human-object detection and interaction classification , namely CDN , to mine the benefits of two-stage and one-stage HOI detectors . Our CDN disentangles the original set-based one-stage optimization function into two cascade decoders . Firstly , we predict human-object pair by P ( bh , bo|V ) . Secondly , we apply an isolated decoder to predict the action category by P ( a|V , bh , bo ) . More details are in the following . 3 Method . In this section , we will present a detailed introduction to the pipeline of our proposed CDN . In section 3.1 , we present an overview of our framework and briefly introduce the pipeline . In section 3.2 , we introduce the visual feature extractor . The cascade disentangling HOI decoder is introduced in section 3.3 . Section 3.4 introduces a novel dynamic re-weighting mechanism that mitigates the long-tailed problem . The detailed training process and post-processing are discussed in section 3.5 . 3.1 Overview . The architecture of our proposed CDN is illustrated in Figure 2 . Our CDN is organized in a cascade manner with a visual feature extractor . Given an image , we first follow transformer-based detection methods [ 1 , 39 ] to apply a CNN followed by a transformer encoder architecture to extract visual features into a sequence . Then we detect HOI triplets in two cascade decoders . Firstly , we apply the Human-Object Pair Decoder ( HO-PD ) to predict a set of human-object bounding-boxes pairs based on a set of learnable queries . Next , taking the output of the last layer of HO-PD as queries , an isolated interaction decoder is utilized to predict the action category for each query . Finally , the HOI triplets are formed by the output of the above two cascade decoders . 3.2 Visual Feature Extractor . We define the visual feature extractor by combining a CNN and a transformer encoder . Fed with an input image I with shape ( H , W , C ) , the CNN generates a feature map of shape ( H ′ , W ′ , Db ) . Then , Db is reduced to Dc by a projection convolution layer with a kernel size 1× 1 . Next , a flatten operator is used to generating the flatten feature Xv ∈ R ( H ′ ×W ′ ) ×Dc by collapsing the spatial dimensions into one dimension . This flatten feature is then fed into a transformer encoder and the position encoding Epos ∈ R ( H ′ ×W ′ ) ×Dc , which distinguishes the relative position in the sequence Xs ∈ R ( H ′ ×W ′ ) ×Dc . Thanks to the multi-head self-attention mechanism , the transformer encoder produces a feature map with richer contextual information by summarizing global information . The output of the encoder is denoted as global memory with a dimension of Dc .

Summary: The paper proposes a novel HOI detection pipeline which includes a human-object pair decoder and interaction decoder once visual feature is extracted. Both the human-object pair decoder and interaction decoder are modeled by a transformer architecture composed of multiple layers of attention. It also presents some techniques for further improvement such as dynamic re-weighting and pair-wise NMS (PNMS). The proposed method presents superior detection results on common benchmarks, namely HICO-Det and V-COCO.