Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: -- The paper proposes Relational Language-Image Pre-training(RLIP) a pre-training paradigm for Human-Object Interaction (HOI) Detection. This paper posits that this pre-training methodology aims to overcome the gaps left by Object Detection pre-training methodologies that a not fully tailored towards HoI Detection. 

-- The paper presents multiple modules to achieve this pre-training objective. Parallel entity detection and Sequential relation inference (ParSe) proposes a DETR-like training architecture that employs query groups for Subject and Object representation and further query groups conditional on these for modeling Relational representations. To enable contrastive learning during training, the paper proposes  Label Sequence Extension (LSE) that performs out-of-batch-sampling to improve quality of negative samples. Finally, to address label noise and ambiguity in relationship modeling, the paper proposes employing Relational Quality Labels (RQL) and Relational Pseudo-Labels (RPL) modules during training.

-- The paper reports the benefits of the pre-training methodology for HOI detection methods for fine-tuned, zero-shot and few-shot settings.
strengths_and_weaknesses: + The paper leverages the entity descriptions of Visual Genome (VG) datasets efficiently to improve the performance on HOI detection, one of the challenging but relevant problems for scene understanding.  The proposed ParSE method leverages the relational entities with context along with subject and object entities to improve <subject-object-relation> prediction.

+ The paper proposed modules to refine the label noise and semantic ambiguity in the VG dataset using the RQL and RPL modules which is a relevant problem for VG dataset.


+ The paper presents strong results for the zero-shot HOI detection over other SoTA approaches demonstrating the effectiveness of pre-training method for zero-shot detection. The paper also shows the contribution of each of the modules through ablation studies showcasing the contributions of each of the modules.

- The proposed method does not show a strong improvement over the CDN method for fully-supervised HOI detection method for both VCOC and HICO-DET datasets. Infact, the performance of the method goes down when including COCO dataset for RLIP compared to just the OD pre-training method using ParSE indicating overfitting of the pre-training method when including COCO.

- The improvement for zero-shot setting could be down to the strong prior in the distribution provided by the VG dataset since the zero-shot formulation is based on OOD rather Unseen setting. Including another dataset with triplet entities of Subject, Object, Relation might showcase the robustness of the method to different data distributions.

- The method ranks low in novelty since it reuses many of the existing HOI detection methodologies and adapts the same for pre-training using VG dataset.
questions: -- Did the authors try any dataset other than VG with similar entity description to showcase the robustness of the model?

-- The authors could try to include more qualitative examples in the supplementary material to showcase success and failure cases.

-- There are couple of small changes that could be made in writing

    - #L57 'negatives samples'  -> 'negative samples'
    - #L144 'We next similarly' -> Next, we similarly
limitations: -- There are no suggestions on the societal impact front --
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: For Human-Object Interaction (HOI) detection, the authors propose Relational Language-Image Pre-training (RLIP), a strategy for contrastive pre-training that leverages both entity and relation descriptions. 
To make effective use of such pre-training, they make three technical contributions: 
(1) a new Parallel entity detection and Sequential relation inference (ParSe) architecture that enables the use of both entity and relation descriptions during pre-training; 
(2) a synthetic data generation framework, Label Sequence Extension, that expands the scale of language data available within each minibatch; 
(3) ambiguity-suppression mechanisms, Relation Quality Labels, and Relation Pseudo-Labels, to mitigate the influence of ambiguous/noisy samples in the pre-training data. 
Through extensive experiments, they demonstrate the benefits of these contributions, collectively termed RLIP-ParSe, for improved zero-shot, few-shot, and fine-tuning HOI detection performance as well as increased robustness to learning from noisy annotations.

strengths_and_weaknesses: Strength

- The authors show the effectiveness of the proposed method on various setups such as few-shot and zero-shot setups.

- The proposed ParSe model consistently shows favorable performance compared to the existing methods.

Weakness

- The original contribution of the ParSe model is unclear. In Sec.3, the authors should further emphasize the technical significance of the proposed ParSe model compared to the existing architectures. From the current explanation, it is hard to see the novelty of the proposed model.

- In Table 1, in order to more comprehensively validate the effectiveness of the proposed pre-training method with external data, it would be better to compare with the existing methods when using VG data when using VG data as well.


========== ------- Comments after the rebuttal ------======== 

I have read the other reviewer's comments and the author's rebuttal, which addresses most of my concerns. Therefore, I would like to raise my score.

questions: In Table 1, even without the RLIP paradigm, how do ParSe and ParseD show favorable performance compared to the previous method?
limitations: The authors addressed the limitations of the proposed method and the potential negative social impact of their work.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper aims to adopt the language-image contrastive pre-training techniques to boost the performance and robustness of the Human-object Interaction (HOI) detection task. For this purpose, this paper first modifies the conventional DETR-based HOI detection framework by decoupling detection and interaction classification and disentangling the subject and object queries. Then, this paper converts the entity labels and relation labels into text and embeds them into a latent space, and constructs contrastive pairs in or out of batches. Extensive experiments have been conducted in HICO-Det and V-COCO datasets from regular, few-shot, and zero-shot settings.
strengths_and_weaknesses: Strengths:
- The idea and motivation of this paper are generally reasonable.
- The proposed method can be easy to adapt zero-shot, few-shot, and regular HOI detection tasks.
- The writing of this paper is generally clear and the proposed method is simple and easy to follow. 

Weaknesses
- The proposed architecture is not new in the HOI detection area, which combines the CDN [1] and PST [2], and the idea is entirely the same as the proposed architecture GEN in the recent paper GEN-VLKT [3]. Though GEN-VLKT is a CVPR 2022 paper, it has been uploaded in Arxiv and released code in Mar. 2022. Thus the author may present a detailed discussion with this paper.

- Limited Application Scenarios. The proposed pre-training techniques still require relation triplets annotations, which are expensive to obtain.

- Missing details about OD+VG. The DETR requires a long time to converge, so the author should report the number of epochs to pre-train the DETR in VG. The significant performance gap between OD and RLIP with VG may come from the insufficient pre-training for OD+VG.

- Missing important baseline. The OD+VG baseline only trains a detector in VG with the detection annotation, but RLIP adopts the relation and detection annotations in the VG simultaneously. Thus a fair baseline is to directly to pre-train the ParSe with the VG visual relationship detection based on the COCO pre-trained model.

- An additional large-scale dataset with a minor improvement. Equipped with COCO+VG pre-training, the performance only improved by 1.05 mAP (31.79->32.84).

[1] Mining the Benefits of Two-stage and One-stage HOI Detection. NIPS 2021.
[2] Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries. ICCV 2021.
[3] GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection. Arxiv Mar. 2022.
questions: - My main concerns and questions lie in the 'Missing important baseline' in the weaknesses. The author should provide the results with the mentioned setting for a fair comparison to verify the effectiveness of the proposed RLIP.

- Confused by the OD+VG performances. VG is a large-scale visual relationship detection dataset, but the performance has dropped a lot when pre-training in this dataset. A good choice may be to train DETR in VG first and then train in COCO because the object labels of HICO-Det are the same as COCO. A detailed analysis of VG should be included.
limitations: The authors provided limitations in supplementary materials.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposes a visual-language pre-training method for HOI detection. Following MDETR and CDN, a pipeline named ParSe is proposed to implement contrastive learning upon humans, objects, and relations. Correspondingly, a pos-neg language labels scheme and a language ambiguity suppression method are proposed for the pre-training. On commonly-used benchmarks, the proposed method is compared with SOTAs on both supervised and zero-shot settings.
strengths_and_weaknesses: Pros:
+ The HOI contrastive learning is an interesting topic and would advance the HOI detection, considering the possibility of open-vocabulary visual-language learning.

+ Some reasonable model designs are proposed to handle the pos-neg label scheme, language label processing, etc.

+ Extensive ablation studies are conducted and presented. Some analysis is useful for follow-ups.

+ Borrowing the idea from MDETR into the v-l HOI contrastive learning is non-trivial.

Cons:
- First, is the novelty of the first contribution. As mentioned by the authors too, they follow CLIP, CDN, and MDETR to build the ParSe model. Though under the v-l setting, few novel designs are proposed to bring new insights. 

- Second, the experiment. In Tab 1, the comparison between methods is mainly within the setting using Res-50 which is reasonable, but for CDN, some results are missing: CDN-B with Res-50: 27.55 33.05 31.78 (which is comparable with the proposed method 26.36 33.41 31.79 in COCO as PT data); and many previous works are also not compared here. If compared with the above result of CDN, it seems marginal improvement is achieved. Though performance does not mean all, considering the similar structure of the proposed method with CDN, concern raises. Meanwhile, if using Res-101 and Swin-tiny, how would ParSe perform, and does it have an advantage compared with CDN and QAHOI as the same series methods. Moreover, ParSe can use extra data in the pre-training, but few improvements are achieved. 

- In the discussion of the first contribution, in fact, many previous methods (iCAN, TIN, etc) adopted the separated representations for a human, object, and relations. Even using a transformer, the differences and similarities should also be discussed. And CDN is another case, which is followed by this work. Thus, I suggest revising the part about the first contribution.
As for the relation contrastive learning, there are some recent works like Contrastive Visual and Language Translational Embeddings for Visual Relationship Detection, Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationships (just for your information, does need to discuss as they are too new).
L84: open-vocabulary recognition remains underexplored. This point is also open to discussion, as VCL, FCL,  Detecting Unseen Visual Relations Using Analogies, etc. also have considered open h-v-o scenarios.
L88: rendering it suboptimal for RLIP. Why, please give a more detailed discussion.

- Method part is somewhat hard to follow as the dense information presented is a short section. And many adopted methods are just given a citation without a brief introduction, which makes the reading interrupted.

- Prompt diversity needs a detailed analysis. 

- Is the zero-shot setting fair (tab 2) as the proposed method can use extra data and the other method do not? Please clarify this.

Typo: L63 Parse --> ParSe
questions: 1. Efficiency of the sequential h-o pairing? 

2. How similar of the pos and neg prompts? Need an analysis.

3. As the pre-training using VG, many zero-shot relations of HICO-DET maybe not zero-shot no more, as there may be many similar and even same relations in VG for training. So a detailed analysis should be taken to probe the **real "zero-shotness"**.

4. In Eq. 5, is the Euclidean distance the best choice? How about the others like cosine distance?
limitations: N/A.
review_title: null

Meta-review: metareview: This paper proposes a free-form relational language-image pretraining for HOI detection which demonstrates advantageous performances on zero-shot and few-shot settings. All reviewers give consistent positive scores after the discussion phase. The authors have added more experiments on COCO, different backbones, and pretraining datasets. And additional quantitative and qualitative analyses to claim motivations are also presented. Given the good insights and performance of the proposed RLIP, the meta-reviewers thus recommend to accept this paper.
confidence: Certain
Decision: Accept