A Plug-and-Play Method for Rare Human-Object Interactions Detection by Bridging Domain Gap

Authors: lijun  zhang
wei  suo
peng  wang
yanning  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-01 03:00:59

Abstract: Human-object interactions (HOI) detection aims at capturing human-object pairs in images and corresponding actions. It is an important step toward high-level visual reasoning and scene understanding. However, due to the natural bias from the real world, existing methods mostly struggle with rare human-object pairs and lead to sub-optimal results. Recently, with the development of the generative model, a straightforward approach is to construct a more balanced dataset based on a group of supplementary samples. Unfortunately, there is a significant domain gap between the generated data and the original data, and simply merging the generated images into the original dataset cannot significantly boost the performance. To alleviate the above problem, we present a novel model-agnostic framework called Context-Enhanced Feature Alignment  (CEFA) module, which can effectively align the generated data with the original data at the feature level and bridge the domain gap. Specifically, CEFA consists of a feature alignment module and a context enhancement module. On one hand, considering the crucial role of human-object pairs information in HOI tasks, the feature alignment module aligns the human-object pairs by aggregating instance information. On the other hand, to mitigate the issue of losing important context information caused by the traditional discriminator-style alignment method, we employ a context-enhanced image reconstruction module to improve the model's learning ability of contextual cues. Extensive experiments have shown that our method can serve as a plug-and-play module to improve the detection performance of HOI models on rare categories.

Main paper: 
 introduction: 
 The goal of Human-Object Interaction (HOI) detection is to localize humans and objects in an image and recognize their interactions. The interactions can be represented by a triplet <Human, Verb, Object>. HOI enables machines to understand human activities at a fine-grained level, which is an important step towards high-level advanced visual reasoning [16, 28, 39] and scene understanding [29, 37, 41] . However, real-world data have a long-tailed distribution such as human riding giraffe, which will lead to the presence of long-tail phenomena in HOI datasets. Long-tail datasets result in mediocre performance of HOI models on rare categories. To address the long-tail problem, existing methods can be divided into two paradigms [1, 13, 44] . The first category is based on training strategies [11, 33, 44] , such as resampling and retraining techniques. Resampling and retraining are achieved by changing the sampling strategy for long-tail data or adjusting the loss function. The other research line focused on combinatorial learning [1, 13, 55] . They utilize the concept of combinatorial learning to recompose human-object pairs and interaction between different HOI instances as new training samples. In this way they promote knowledge generalization across HOI classes and form new triplets to mitigate the long-tail problem.Although these methods have obtained great progress, they have not fundamentally addressed the long-tail issue caused by data imbalance. Recently, Artificial Intelligence Generated Content (AIGC) has made significant advancements in generating highly realistic images. This has opened up possibilities for using AI-generated data as a replacement for real data. Therefore, a straightforward idea is to utilize generative models to synthesize rare categories in the HOI dataset. Unfortunately, as shown in Fig. 1 (a), we find that simply merging the generated images into the original dataset and training the model does not significantly improve the performance of the model. [4, 40] have made some initial explorations about this, they alleviated the problem by designing different rules to filter out low-quality images that exhibit substantial deviation from the original dataset. However, these approaches lack scalability across different tasks and suffer from large human subjectivity. In order to explore the critical reasons, as depicted in Fig. 1 (b), we use t-SNE [31] to analyze the feature distribution between original data and generated data. From the visualization results, it can be found that there are significant distribution differences between the generated data and the original data. In other words, there exists a considerable domain gap between the original and generated data.To address the aforementioned issues, we propose a novel yet simple feature alignment framework called Context-Enhanced Feature Alignment (CEFA) module, which aims to align the generated data with the original data at the feature level and bridge the domain gap. On one hand, most existing feature alignment methods based on transformer architecture [7, 14, 36, 47] mainly align encoder and decoder features through the discriminator, without considering the instance information. While in the Human-Object Interaction tasks, the instance information of human-object pairs is critical for successful task execution. Since HOI triplets are predicted through the features of the decoder, we aim to pay more attention to instance alignment during the decoder feature alignment process. Considering that tokens in the decoder can often be repetitive or even erroneous, we propose a graph-based prototype instance alignment method. This approach views the high-scoring tokens as prototypes and constructs a graph network to align the human-object pairs by aggregating instance information. On the other hand, some researches [10, 21] have indicated that the above discriminator-style alignment method inevitably leads to losing important context clues. These context clues provide important contextual information to identify human-object interactions. For example, when the background is snow, the model should be more inclined to predict "people holding skis". Based on the above insights, we propose a context-enhanced image reconstruction module, aiming to improve the model's learning ability of contextual cues. Benefitting from the above two modules, as shown in Fig. 1 (a) and (c), our model effectively extracts more domain-independent features and bridges the domain gap.In summary, the primary contributions of our paper are as follows:• We propose a novel model-agnostic method CEFA which can effectively bridge the domain gap between the generated and original domains. • We build an instance feature alignment module to align the human-object pairs by aggregating instance information. Meanwhile, we design a context-enhanced image reconstruction module to improve the model's learning ability of contextual cues. • The proposed CEFA is a plug-and-play module that can be conveniently applied to existing HOI models and boost the performance of rare categories. 
related work: 
 Long tail solution in HOI detection. To address long-tail phenomena in HOI, researchers have proposed various methods [2, 12, 15, 34, 38, 54] , which can be summarized into two categories: training strategy-based methods and combinatorial learning-based methods. Training strategy-based methods [11, 33, 44] primarily improve the training process. They attempt to alleviate the long-tail problem by adjusting loss functions, sample sampling strategies, class balancing techniques, etc. Combinatorial learning-based methods [13, 35, 55] aim to enhance the performance of HOI models by recombining different representations of human-object pairs and interaction, forming new triplets to mitigate the long-tail problem. While existing methods have not fundamentally addressed the long-tail issue caused by data imbalance. AIGC offers a promising solution by using generative models to synthesize rare classes in the HOI dataset. Recently [4, 40] has made some initial explorations, while they suffer from the problem of large differences between the generated images and the original images. They addressed this by devising a set of rules to filter the generated images. These approaches lack scalability across different tasks and have relatively large human subjectivity. To this end, we propose CEFA to align the generated data with the original data at the feature level.Unsupervised domain adaptation. To enhance the domain adaptation capabilities of the model, researchers have proposed various unsupervised domain adaptation techniques [5, 17, 42, 49] . Next, we will introduce Unsupervised domain adaptation using object detection as an example. For object detection, the goal of unsupervised domain adaptation is to narrow the domain gap between the source and target domains so that the source data can be utilized to train a better detector for the target data. Most existing domain adaptive detectors [7, 14, 26, 36, 47] employ a teacher-student model to generate pseudo-labels and utilize adversarial learning to encourage the model to extract domain-invariant features, thereby improving domain transferability. Among them, based on the transformer architecture [7, 14, 27, 36, 47] primarily align feature extracted by CNN and token sequences from the transformer encoder-decoder. For instance, [36] aligns features at global and local levels to enhance performance in domain-adaptive object detection. While recent studies [10] have shown that the above discriminator-style alignment method will cause the model to focus on domain-invariant features while losing important context clues. We propose a context enhancement module to alleviate this problem. 
propose method: 
 In this section, we introduce our Context-Enhanced Feature Alignment (CEFA) framework, which aims to reduce the distribution difference between the generated domain and the original domain, thereby alleviating the HOI long-tail problem. As shown in Fig. 2 , the CEFA comprises an "Instance Feature Alignment Module" and a "Context Enhancement Module".Since our CEFA is a plug-and-play paradigm that can be applied to the previous one-stage based HOI model, we begin with a detailed introduction to the foundational architecture of the HOI model in Section 3.1. Subsequently, in Section 3.2, we expound on the instance feature alignment module, which aligns the humanobject pairs by aggregating instance information. In Section 3.3, we describe the context enhancement module, which employs a masked image reconstruction branch to strengthen the model's ability to capture contextual cues. Finally, in Section 3.4, we provide a comprehensive introduction to the loss function.In this section, we first introduce the formal definition of the HOI detection problem. Given a human-object image 𝐼 , the model is required to predict a set of HOI triplets 𝑆 = {(𝑏 ℎ 𝑖 , 𝑏 𝑜 𝑖 , 𝑣 𝑖 ), 𝑖 ∈ {1, 2, ..., 𝑁 }}, where 𝑏 ℎ 𝑖 , 𝑏 𝑜 𝑖 and 𝑣 𝑖 denotes a human bounding-box, an object bounding-box and their corresponding action category, respectively. Since our CEFA is a plug-and-play paradigm, we would first introduce the basic HOI model (e.g., [44] ).A basic HOI model can be divided into three components: visual feature extractor, human-object pair decoder, and relationship decoder. The visual feature extractor, consisting of a CNN and a Transformer encoder, is responsible for extracting image features and outputting encoder features 𝑋 𝑒 ∈ 𝑅 (𝐻 ′ ×𝑊 ′ ) ×𝐷 𝑐 . The encoder features 𝑋 𝑒 and the human-object pair query 𝑄 𝑖𝑛𝑠 ∈ 𝑅 𝑁 𝑞 ×𝐷 ℎ𝑞 are fed into the human-object pair decoder to output 𝑋 ℎ𝑜 ∈ 𝑅 𝑁 𝑞 ×𝐷 ℎ𝑑 , which is used to detect the position of humans and objects. Then the encoder features 𝑋 𝑒 , the human-object pair decoder outputs 𝑋 ℎ𝑜 are fed into the relationship decoder. The relationship decoder outputs 𝑋 𝑣𝑒 ∈ 𝑅 𝑁 𝑞 ×𝐷 𝑣𝑒 to detect relationship categories for the corresponding pairs of humans and objects.Although the above basic pipeline has achieved promising results in HOI detection. Unfortunately, due to the natural bias from the real world, existing methods mostly struggle with rare humanobject pairs and lead to sub-optimal results. Recently, with the development of the generative model, a straightforward approach is to build a more balanced dataset with a group of supplementary samples. However, as shown in Fig. 1 (a), we find that simply merging the generated images into the original dataset cannot significantly improve the performance of the model. In order to reveal the potential reasons, as depicted in Fig. 1 (b), we use t-SNE [31] to analyze the distribution between original data and generated data. From the visualization results, it can be found that there exists a considerable domain gap between the original and generated data. To bridge the domain gap and align features, we propose the instance feature alignment module and the context enhancement module. Next, we will provide a detailed introduction to these modules.Many previous domain alignment techniques [7, 14, 36, 47] based on the transformer architecture primarily align features extracted by CNN and token sequences from the transformer encoder-decoder. Most of them feed both CNN features and all tokens into discriminators for alignment. For CNN features and transformer encoder features, similar as in SFA [36] , we feed them into the discriminator for alignment, with the following loss: where 𝑠𝑟𝑐 represents the images from the original domain and 𝑔𝑒𝑛 represents the images from the generated domain, 𝐷 𝑏 and 𝐷 𝑒 denotes the domain discriminator. 𝑋 𝑏 and 𝑋 𝑒 denotes the CNN features and transformer encoder features.EQUATIONIn the Human-Object Interaction tasks, the instance information of human-object pairs is critical for successful task execution. Since HOI triplets are predicted through the features of the relationship decoder, we aim to concentrate more on instance alignment during the relationship decoder feature alignment process. Considering that tokens in the relationship decoder can often be repetitive or even erroneous, feeding all relationship decoder tokens into the discriminator for alignment would lead to suboptimal results. Therefore, as shown in Fig. 3 , we propose a prototype instance alignment module to aggregate instance information.For the output of the relationship decoder, due to most of the predictions learned by the token are redundant or even incorrect, we select the top 𝑘 sets of relationship tokens with the highest scores as human-object instances. Following [53] , each encoding can be viewed as a prototype, we consider the selected instances as prototypes.Considering the presence of redundant tokens, in order to better aggregate instance information, we aim to enable regular tokens to transfer instance information into the prototypes. Given the convenience of graph networks in specifying information propagation rules, we construct a graph network to aggregate instance information. Specifically, we construct a graph G = (V, 𝐴, 𝑋 ), where V = {𝑣 0 , 𝑣 1 , . . . , 𝑣 𝑁 𝑞 } represents the set of 𝑁 nodes, 𝐴 ∈ R 𝑁 ×𝑁 represents the adjacency matrix of the graph, and 𝑋 ∈ R 𝑁 ×𝐷 𝑣𝑒 represents the node feature matrix. Each graph node represents a token feature. We classify the nodes representing prototypes as prototype nodes, while the rest are referred to as regular nodes. To propagate information from regular nodes to prototype nodes for instance aggregation, there are edges connecting prototype nodes with regular nodes and other prototype nodes. There are no edges connecting regular nodes to regular nodes.The Graph Convolutional Network (GCN) [46] can effectively capture the local and global context of nodes in the graph. By facilitating information transfer between nodes, it enables effective representation learning. So we utilize GCN for feature propagation among the nodes. Following [46] , the formula for the feature propagation is as follows:EQUATIONEQUATIONwhere 𝐴 represents the adjacency matrix, 𝑋 represents the node feature matrix, 𝑃 represents the set of prototype indices.We extract the features of the prototype nodes as the instance features 𝑋 𝑝 and input them into the discriminator for feature alignment. The instance loss is defined as follows:EQUATIONwhere 𝐷 𝑑𝑒𝑐 denotes the domain discriminator.In summary, the adversarial optimization objective is formulated:EQUATIONwhere 𝑇 , 𝐷 denotes the HOI detector and discriminators respectively. Gradient Reverse Layers(GRL) [5] is adopted for min-max optimization.The aforementioned feature alignment module aligns features from three different sources: CNN features, transformer encoder features and instance features. While studies [10, 21] indicate that the above discriminator-style alignment method will cause the model to focus on domain-invariant features at the expense of losing important context clues. In the HOI task, context clues can provide rich scene information to help the model better identify human-object interactions. So we propose a context-enhanced image reconstruction module to improve the model's learning ability of contextual cues.As shown in Fig. 2 , we randomly mask the generated image 𝐼 𝑔𝑒𝑛 at a certain ratio 𝜎 to obtain 𝐼 𝑚𝑎𝑠𝑘 . The original image 𝐼 𝑠𝑟𝑐 and the masked generated image 𝐼 𝑚𝑎𝑠𝑘 are separately fed into the encoder to obtain the features 𝑋 𝑠𝑟𝑐 and 𝑋 𝑚𝑎𝑠𝑘 . Considering that the generated images share the same semantic information as the original images with the same labels, we use the features of the original images as auxiliary information to aid the recovery process of the masked images. Firstly, we fed 𝑋 𝑠𝑟𝑐 and 𝑋 𝑚𝑎𝑠𝑘 into the conditional controller. The role of the conditional controller is to learn the importance of different features from the original image for the image restoration process. It is composed of the decoder layer from the Transformer [32] . During this process, the 𝑋 𝑠𝑟𝑐 serves as the query as well as 𝑋 𝑚𝑎𝑠𝑘 serves as the key and value. The output of conditional controller is denoted as 𝐷 𝑠𝑖𝑔𝑛 . Secondly, Considering that not all features in the original image contribute to the recovery of the generated image, we select the 𝐷 𝑠𝑖𝑔𝑛 that are greater than 0.5 using a gate mechanism. The final original image feature used for conditionally control 𝑋 𝑠𝑟𝑐_𝑐𝑜𝑛𝑑 is obtained by element-wise multiplication between 𝑋 𝑠𝑟𝑐 and filtered 𝐷 𝑠𝑖𝑔𝑛 . In summary, the conditional controller and the gate mechanism work together to filter out the feature tokens irrelevant to image restoration, extracting only the useful features that facilitate the recovery of the image. The above computations can be formulated as:EQUATIONEQUATIONEQUATIONwhere Condition and Gate are conditional controller and gate, ⊙ indicates the element-wise multiplication.We concatenate 𝑋 𝑠𝑟𝑐_𝑐𝑜𝑛𝑑 with 𝑋 𝑚𝑎𝑠𝑘 to obtain the fused features. These fused features are then fed into an auxiliary decoder similar to MAE [9] to reconstruct the original image. It can be formulated as:EQUATIONFollowing [9] , our context-enhanced image reconstruction loss 𝐿 𝑐𝑡𝑥 is calculated by computing the Mean Squared Error(MSE) loss between the generated image and the reconstructed image. The calculation formula is as follows:EQUATIONwhere 𝐼 𝑖 𝑟𝑒𝑐𝑜𝑛 represents the reconstructed image patch and 𝐼 𝑖 𝑔𝑒𝑛 represents the generated image patch.Our model has two types of inputs: generated images and original images. We denote 𝐿 𝑠𝑢𝑝 as the supervised HOI loss of original images and 𝐿 𝑢𝑛𝑠𝑢𝑝 as the HOI loss of generated images. 𝐿 𝑠𝑢𝑝 is computed using labeled original images and 𝐿 𝑢𝑛𝑠𝑢𝑝 is calculated using generated images with pseudo-labels. The loss for the original images 𝐿 𝑠𝑟𝑐 includes the supervised HOI loss and the discriminator loss, defined as:EQUATION)The loss for the generated images 𝐿 𝑔𝑒𝑛 includes the HOI loss, the discriminator loss, and the context reconstruction loss, defined as:EQUATIONTherefore, the total loss is given by:EQUATION) 
experiments 4.1 experimental setting: 
 Datasets. Following previous work, we evaluated our model on two common benchmarks, HICO-Det [3] and V-COCO [8] . HICO-Det consists of 47,776 images, with 38,118 images used for training and 9,658 images used for testing. It includes 80 object categories and 117 action categories, forming a total of 600 HOI triplets. Among these 600 HOI categories, any category with fewer than 10 training instances is defined as Rare, while the remaining categories are Implementation details. We use Blip-diffusion [20] to generate images with the template "a photo of human verbing object", where verbing represents the present progressive form of the action in the HOI triplet. The threshold for pseudo-label with the model is set to 1.4 for GEN-VLKT and HOICLIP benchmark models, and 0.4 for the CDN benchmark model. During the training phase, we fine-tune the pre-trained weights of the existing HOI model and keep the human-object pair decoder fixed. The learning rate for the backbone in the CDN benchmark is set to 1e-5, while the rest of the learning rates are set to 1e-4. For the GEN-VLKT and HOICLIP benchmark, the backbone learning rate is set to 1e-6, and the rest is set to 1e-4. The mask ratio 𝜎 is 0.8, and the number of prototype nodes 𝑘 is set to 6. We conduct all the experiments on 8 Tesla V100 GPUs, using a batch size of 12 per GPU.Evaluation. We evaluated our model using mean Average Precision (mAP). The HOI triplet predictions are considered true positives when they meet the following criteria: 1) the predicted human and object bounding boxes have an IoU greater than 0.5; and 2) both predicted HOI categories are accurate. For HICO-Det, we evaluated three different category sets: the complete set of 600 HOI categories (Full), the 138 HOI categories with fewer than 10 training instances (Rare), and the remaining 462 HOI categories (Non-Rare). For V-COCO, we also evaluated three different category sets: the complete set of 29 action categories (Full), the 4 action categories with fewer than 300 training instances (Rare), and the remaining 25 action categories (Non-Rare).To evaluate the effectiveness of our method, we conducted combined experiments with three representative methods. Firstly, we performed experiments on the HICO-Det dataset, the results are shown in Table 1 . Since the proposed method is a plug-and-play module, our method is combined with CDN, GEN-VLKT, and HOICLIP to achieve performance improvements. Specifically, when combined with CDN, our method achieved a relative improvement of 4.05% on rare categories. When combined with GEN-VLKT and HOICLIP, our method achieved a +1.49 mAP and +1.59 mAP improvement on rare categories. These results demonstrate the effectiveness of our method in handling rare class HOI tasks. We have also noted that merely merging the generated data with the original dataset for training does not lead to a direct and intuitive improvement in performance. This confirms that the domain gap between the original and generated data causes the model to deviate from the optimal solution during the fine-tuning process, thereby impeding the enhancement of performance. After applying our method, there is a significant increase in performance, which further attests to the effectiveness of our approach. Additionally, the compatibility of our method with various other methods also proves its excellent extensibility. We also compared our method to previous state-of-the-art (SOTA) approaches, it can be found that our method outperforms these works by a big margin. Next, Table 2 presents the performance of our method on the V-COCO dataset. The results showed significant improvements over the original models. Specifically, when CDN is combined with our method, we achieved a +1.02 mAP improvement for all categories and an impressive +2.23 mAP improvement on rare categories. When combined with GEN-VLKT and HOICLIP, we achieved a +1.64 mAP and +1.38 mAP improvement on rare categories.Model components ablation. In this section, we conduct several ablation studies on the HICO-Det to systematically evaluate the contributions of different model components. As shown in Table 3 , the "Instance" and "Context" represent instance feature alignment module and context enhancement module, respectively. Specifically, "Instance" indicates the process of aligning human-object pairs by aggregating instance information. "Context" represents using context-enhanced image reconstruction module to improve the model's learning ability of contextual cues. Due to HOICLIP [25] achieving state-of-the-art results in previous methods, we selected it as the baseline for comparison. As shown in rows 1-2 of Tabel 3, we follow the baseline and add an instance feature alignment module to align features. We can observe that the rare class mAP achieves an improvement of 1.06, which demonstrates that our instance feature alignment module is capable of effectively aligning human-object pairs, thereby promoting domain alignment. In addition, in row 3 of Table 3 , we follow the baseline and add the context enhancement module separately to extract important contextual information. The context enhancement module resulted in a relative improvement of 3.91% on rare categories, which shows that the proposed context enhancement module can effectively learn important contextual cues and alleviate the loss of contextual information.Alternative model settings. In Table 4 , we exploit some alternative experiments about the different model settings on the HICO-Det dataset. To validate the effectiveness of our proposed conditional controller in the context enhancement module, we removed it and used the original mask autoencoder to reconstruct the images. The results in the first row indicate that adding the original image as a conditional control can further guide the model to better reconstruct images and promote the learning of contextual clues. In the second row of the table, we replaced the graph convolutional network with a transformer model in the instance feature alignment module. The results show that the transformer model is not wellsuited for aggregating instance information. The main reason is that the transformer model does not consider the special relationship between prototype nodes and regular nodes. In the 3-4 rows of Table 4 , we replaced the cross-attention with MLP and self-attention in the conditional controller, respectively. The results indicate that MLP cannot effectively serve as the conditional controller because it lacks the ability to fuse and control information compared to cross-attention. We observed that using both cross-attention mechanisms and self-attention mechanisms as conditional controllers yielded excellent performance, which proves that these two mechanisms can extract meaningful conditional control signals.As shown in rows 5-6 of Table 4 , we replaced the bidirectional graph with fully connected and directed graphs. Directed graphs refer to there are edges connecting from ordinary nodes to prototype nodes. In row 5 of Table 4 , our bidirectional graph outperformed the fully connected graph by 0.92 mAP, indicating the importance of considering the special relationships between prototype and ordinary nodes. Row 6 indicates a slight impact of relationship propagation from prototype nodes to ordinary nodes.Generate quantity settings. As shown in Fig. 4 , we studied the impact of the number of generated images on the HICO-Det. We find that the performance of rare classes improves with an increasing number of generated images, but does not increase after the number of generated images exceeds 100. In order to maintain a balance between performance and resource consumption, we decided to generate 50 images per rare class.To more intuitively demonstrate the effectiveness of our approach, we visualize some examples of experimental results, as shown in Fig. 5 . In the first row, we show the ground truth labels. The second row displays the predictions of the previous state-of-the-art method HOICLIP [25] . The third row represents the predictions of our CEFA model. In the figure, blue bounding boxes represent person, and green bounding boxes represent objects.By observing the 1-2 columns, we find that the HOICLIP exhibits errors in the localization and classification of humans and objects, which subsequently affects the prediction of relationship interactions. In contrast, our model achieves more accurate localization and classification, enabling the correct prediction of relationship interactions. Furthermore, by observing the the last three columns, we can observe that even the HOICLIP correctly detects humans and objects, errors can still occur due to failed relationship predictions. While our model is capable of improving this situation.We present the distribution of features extracted by HOICLIP. As shown in Fig. 1 (b), the generated image and original image features can be easily separated by domain. In contrast, our proposed CEFA model is capable of learning domain-invariant features during the feature extraction phase. As illustrated in Fig. 1 (c ), after processing with our approach, the image features from different domains become significantly more aligned and are not easily distinguishable. 
conclusion: 
 In this paper, we introduce a model-agnostic framework that can be conveniently applied to existing HOI models and boost the performance of rare categories. Specifically, we propose Context-Enhanced Feature Alignment (CEFA) module to align the generated data with the original data at the feature level. The CEFA helps the model overcome the domain gap, allowing the use of generated data for practical training. Experimental results demonstrate that our CEFA achieves promising performance in terms of accuracy and scalability. We expect that this approach will provide a new paradigm for our community. 
 
 Appendix:  


Decision: The initial reviews lean slightly and consistently towards accepting this paper. After the efficient rebuttal, two reviewers upgraded their rates from borderline accept to weak accept. The paper proposes a model-agonistic framework to align the domain gap between the generated data and the original data for rare HOI detection. It shows strong empirical results indicating the effectiveness of the proposed method. The AC thus would like to recommend it for publication.

Reviews: review: paper_summary: paper_summary: This paper introduces a model-agonistic framework to align the domain gap between the generated data and the original data for rare HOI detection. Empirical results indicate the effectiveness of the proposed method for HOI detection.
strength_weakness: strength_weakness: (1) The idea of prototype instance alignment is interesting and reasonable. 
(2) Experiments are extensive and demonstrate the effectiveness of the method.
(3) The paper is generally well-written and in good quality.
limitations: limitations: (1) It is not quite clear how does the generated images be produced based on the original image. How to guarantee the generated image is in the same domain of the original one? How about the quality of the generated image?

(2) In the Instance Feature Alignment Module, how does the relationship decoder model the relationships of human and object? It seems the relationship decoder only takes the human feature and object feature as inputs, without the information of interactions, which would make it difficult to model relationships between human and object.

(3) The paper focus on modeling rare HOIs, how does the proposed method deal with the challenge of rare events?
paper_summary: paper_summary: The authors propose to alleviate the long-tail problem in HOI detection. Firstly, the authors found that the generated images have a distributional shift with the real world data. Then, the authors propose two modules to align their features at the instance level. The authors conduct extensive experiments on benchmark dataset. The results showing that the proposed method can indeed improve the performance on various base models, especially on tail classes.
strength_weakness: strength_weakness: + The problem studied in this paper is important.

+ The observation that generated data has a different distribution with real-world data is interesting and new to this area.

+ The proposed method is well-motivated and effective as demonstrated by experiments.

+ The ablation studies provide in-depth analysis of the proposed method.
limitations: limitations: Any idea on the domain gap between real world data and generated data? Is it related to the distribution shift between HICO-DET and LAION? On the other hand, does the generated image contain only one person? Does this also contribute to the distribution shift? It would be more insightful if the authors could further discuss this.

Following the above question. I wonder is there any difference when using different generative models? Will the domain gap be different?


Some related references are missing:

[1] Lin et al. Learning from Easy to Hard Pair: Multi-step Reasoning Network for Human-Object Interaction Detection. ACM MM 23

[2] Wang et al. Distance Matters in Human-Object Interaction Detection. ACM MM 22
paper_summary: paper_summary: This work tackles the human-object interaction (HOI) detection task on 2d images, which is essentially an image-based unimodal detection task similar to object detection. As multi-modal diffusion models are adopted in the proposed method to enhance the performance of the HOI detection model, I think the topic of this work is moderately (but not strongly) related to the MM society.

This work use the generated images from diffusion models to augment the HOI detector, especially on the tail classes with fewer natural samples. To tackle the significant domain gap between the generated images and the original samples, this work proposed a method with two components. The first is to align the instance features from the original samples and the generated samples. The second is to reconstruct the generated samples from image patches, to strengthen the learning on contextual cues.
strength_weakness: strength_weakness: - The major problem to tackle in this work is valid and somewhat interesting. Two previous works try to benefit HOI detection models with generated samples, but they do not notice the domain gap between the generated and the original ones, which is empirically a trouble for this approach.
- Moderate experimental improvement on rare (tail) classes. This shows the method is effective on the tail classes, as they claim to solve.
limitations: limitations: - The relationship between the motivation and the proposed modules needs more clarification.
    - The second component, the context enhancement module, seems not strongly related to the overall motivation. This work aims to tackle the problem of domain gaps between the generated and the original samples. Why does the model need to strengthen the learning on the contextual cues? This might need more clarification. If the method is to solve the problem of losing context due to the first component, then the experimental results in Table. 3 seems not supporting this, as adding the second to the first brings limited improvement over using only one component.
    - Similarly, the relationship between the first component and the motivation, though can be understood, is not very clearly stated in the paper.
- Limited novelty and importance of the technical design for the first component. The GNN-based feature fusion in PIAM has been adopted by many HOI detectors, and the difference and necessity for such design should be clarified. Besides, Table 4 shows the specific choice of the design has little effect on the overall performance.
- The proposed method is not so "plug-and-play" and claimed. Firstly, as shown by Fig.2, it is only applicable to HOI detection methods with CNN+transformer+HOI detector+relationship decoder, which constitutes only one part of the different frameworks of HOI detectors. Secondly, the method introduces multiple auxiliary losses and additional networks. This limits the ability to transplant it onto different baselines.
- Mistakes in writing. I understand that some typos or grammatical errors in the main sections are inevitable in the paper and there are no need to mention them, but this paper contains several mistakes in the *abstract*, which should be checked multiple times before submitted. For example, "Unfortunately, there is a significant domain gap between the generated data and the original data, simply merging the generated images into the original dataset cannot significantly boost the performance" is grammatically wrong. "On one hand, Considering the crucial role ..." contains obvious typos.
paper_summary: paper_summary: This work addressed the domain gap (between generated images and training set) of human-object interaction detection, especially for the rare classes. A model-agnostic framework to align the generated data with the original training set a the feature level. The model is plug-and-play and the improvement compared to naive data augmentation is of good margin.
strength_weakness: strength_weakness: + This work leverages the generative AI model to augment training data for HOI detection tasks. The authors observed that naive augmentation does not work., due to image quality and domain gap.
+ A novel framework (i.e., CEFA) was proposed to perform feature-level alignment to bridge the domain gap, which considers the instance alignment through a graph-based prototype instance alignment method. In addition, a context-enhanced image reconstruction module is proposed for learning the contextual cues.
+ The proposed CEFA is model agnostic and can benefit many HOI detection approaches.
+ The proposed CEFA shows good improvement on rare classes on three HOI baselines. The authors also provide a number of ablation studies to support the claims.
limitations: limitations: - Can the author provide an intuition of the context enhancement module? Specifically, how can the proposed method leverage more context clues using the proposed mechanism?
- For the ablation studies, generate quantity settings, is the number of generated images constant for all classes? Has the author considered a more class-dependent strategy where more images are generated for rare classes and less for the common classes?
- In supplementary material, Section 4, it indicates that the negative predictions are mainly attributed to the incorrect localisation and classification of human-object pairs. This is a STRONG statement, can you author provide evidence that if ground truth object localisation and class label are given, the model can achieve highly accurate results? 
- Tab 3 shows that both the instance feature alignment module and context enhancement module improve the overall HOI detection performance. In this work, the generated samples are fixed and no steps are performed to improve the generated image quality (or diversity). I am curious about which of the modules can better reduce the bias of the learned features that are shown in Fig 1(c).

Others
- In Fig 1's caption, "baseline+Date" --> "Baseline+Data"