RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection
Authors: Hangjie Yuan
Jianwen Jiang
Samuel Albanie
Tao Feng
Ziyuan Huang
Dong Ni
Mingqian Tang
Venue: NeurIPS 2022 Accept (2022)
Keywords: human-object interaction detection
language-image pretraining

Abstract: The task of Human-Object Interaction (HOI) detection targets fine-grained visual parsing of humans interacting with their environment, enabling a broad range of applications. Prior work has demonstrated the benefits of effective architecture design and integration of relevant cues for more accurate HOI detection. However, the design of an appropriate pre-training strategy for this task remains underexplored by existing approaches. To address this gap, we propose $\textit{Relational Language-Image Pre-training}$ (RLIP), a strategy for contrastive pre-training that leverages both entity and relation descriptions. To make effective use of such pre-training, we make three technical contributions: (1) a new $\textbf{Par}$allel entity detection and $\textbf{Se}$quential relation inference (ParSe) architecture that enables the use of both entity and relation descriptions during holistically optimized pre-training; (2) a synthetic data generation framework, Label Sequence Extension, that expands the scale of language data available within each minibatch; (3) ambiguity-suppression mechanisms, Relation Quality Labels and Relation Pseudo-Labels, to mitigate the influence of ambiguous/noisy samples in the pre-training data. Through extensive experiments, we demonstrate the benefits of these contributions, collectively termed RLIP-ParSe, for improved zero-shot, few-shot and fine-tuning HOI detection performance as well as increased robustness to learning from noisy annotations. Code will be available at https://github.com/JacobYuan7/RLIP.

RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection
Hangjie Yuan
Jianwen Jiang?*
Samuel Albanie
Tao Feng"
Ziyuan Huang' Dong Ni Minggian Tang
ZZejjang University
Alibaba Group University of Cambridge
National University of Singapore
Ohj. yuan, dni] edu. en {jianwen. jjw, shisi.ft, ft, mingqian. tmq) Gal haalibbba-ae ietaaiin.cooo com
sma71@cam. ac.uk uk ziyuan. huang@u. nus edu
Abstract
The task of Human-OBe Object Interaction (HOI) detection targets fine-grained visual parsing of humans interacting with their environment, enabling broad range of ap- plications. Prior work has demonstrated the benefits of effective architecture design and integration of relevant cues for more accurate HOI detection. However, the design of an appropriate pre-training strategy for this task remains underexplored by existing approaches. To address this gap. we propose Relational Language- Image Pre-training (RLIP), a strategy for contrastive pre-training that leverages both entity and relation descriptions. To make effective use of such pre-training, we make three technical contributions: (I) new Parallel entity detection and Sequential relation inference (ParSe) architecture that enables the use of both entity and relation descriptions during holistically optimized pre- training; (2) synthetic data generation framework, Label Sequence Extension, that expands the scale language data available within each minibatch; (3) mechanisms account for ambiguity, Relation Quality Labels and Relation Pseudo-Labels, to mitigate the influence of ambiguous/noisy samples in the pre-training data. Through extensive experiments, we demonstrate the benefits of these contributions, collectively termed RLIP-ParSS, for improved zero-shot, few-shot and fine-tuning HOI detection per- formance as well as increased robustness to learning from noisy annotations. Code will be available at httph:t///// pettps:/giitan git.....tbb. com Bate///acoYuuan/// //RIIP
Introduction
Driven by improvements storage, sensors and networking technology, humanity amassing vast archives of image and video data. A significant fraction of this media is human--entric-iit is content focused on humans and their actions. The task of human-object interaction (HOT) detection 4 aims to provide a step towards fine-grained parsing of such content by detecting all possible triplets of the form <human, relation, object> present in visual data. Robust HOI detection has myriad uses for image/video data analysis and represents essential functionality for visual and language applications
such image/video captioning [62 48), image retrieval 27
image synthesis (26] and video action
understanding 24 65).
Given that sustained progress in object detection has yielded increasingly robust systems for detecting people and objects (53 16 10,, key remaining challenge for HOI detection is develop methods capable of generalising to the many possible pairs of interactions between these entities when provided with -oneuxhaustive training data. To tackle this challenge, draw inspiration from recent
'Equal contribution. This work was done when Hangjie Yuan was an intern at DAMO Academy, Alibaba Group, supported by Alibaba Research Intern Program. 1 Correspondiig author.
36th Conference on Neural Information Processing Systems (NeurIPS 2022).

---

term Relational Quality Labels (RQL). To mitigate relation ambiguity, we scheme, which we developments demonstrating that contrastive language- pre-training can induce remarkable generalisation for zero-shot classification tasks [52 25]. These methods perform classification by casting it as retrieval problem, ensuring that the downstream task aligns closely with the pre-training objective. Recent work by Alayrac al. 2 hypothesises that it is this close alignment between the downstream and pre-training objectives that explains why contrastive methods have proven so effective for zero-shot classification. In light of this hypothesis, in this work, we explore whether it is possible to achieve a similarly close alignment between the HOI detection task and its pre-training strategy.
While HOI detection has been widely studied [15
49
29
30 57 66 91, the topic
of designing pre-training to reflect the final task objective remains under-explored. Indeed, a widely adopted strategy [66] 68 57 7 [7] has been to employ object detection pre-training to initialise the parameters of the model responsible for both entity detection and relation inference. However, while suitable for entity detection, such pre-training may be suboptimal for the detection of relations between entities which often requires the model to take account of groups of entities with greater spatial context, rather than individual entities in isolation.
To address this shortcoming of HOI detection, we propose Relational Language-Image Pre-training (RLIP) which tasks the model with establishing correspondences from both entities and relations to free-form text descriptions. By doing so, RLIP endows the model with the ability to perform zero-shot HOI detection Moreover, in contrast to previous pre-training schemes that are limited to predefined finite category lists, RLIP benefits from the rich descriptive nature of natural language supervision.
We encountered three barriers to a naive implementation of RLIP for existing methods: (1) Recent end-to-end HOI detection architectures [57
68 71 66 typically employ joint representations of (some subset of) subject, object and relation triplets. As a consequence, it is difficult to leverage text descriptions for separate humans, objects and relations provided by existing datasets such VG [32) (2) Contrastive pre-training requires negative samples to train effectively, but it is unclear a priori how such negatives should be constructed.
(3) Free-form text descriptions exhibit label noise and semantic ambiguity (since there can be many ways to describe the same concept in the absence of a canonical list of categories), rendering optimisation challenging.
To overcome these barriers, we make several technical contributions in addition to the RLIP frame- work. First, to allow end-to-end contrastive pre-training with distinct descriptions of subsets, objects and relations, propose the Parallel entity detection and Sequential relation inference (ParSe) architecture. ParSe employs DETR like design that allocates separate learnable query groups for subject and object representations, together with an additional set of conditional queries that encode relations. While ParSe enables (and works best with) RLIP, we also find that yields gains for traditional object detection pre-training schemes. To address the second barrier, we synthesise label sequences by extending bn-batch labels with out-of-batch sampling to ensure a plentiful supply of negatives- we term this Label Sequence Extension (LSE). For the third barrier, we exploit cross- modal cues to resolve label noise and relation ambiguity. In particular, to mitigate label noise we use the quality of the visual entity detection phase 337 to assign quality scores to relation-text correspon- dences, approach leverage similarities between labels to propagate relations via term Relational Pseudo-Labels (RPL).
We demonstrate through experiments that relational pre-training outperforms traditional object detection pre-training schemes on comparable data. We further find that the a zero-shot application of our combined approach, ParS- surpasses several existing fine-tuned methods.
2 Related Work
Human-object interaction detection. There a rich body of work on HOI detection. One theme has focused on the development of effective architectures for this task 440
57 9.. A second 29
theme has sought to leverage informative cues ranging from interaction points 40 69). interaction
boxes [ contextualized embeddings (57 30
30), poses [1] 39 and statistical priors 31 66
to external knowledge in the form of language embeddings 61 49 66). However, relational pre-
training and open-vocabulary recognition remains underexplored. The inter-pair transformations of
"Zero-hor in this context refers to HOI detection without fine-tuning (following the terminology of [52). a
formulation that assesses the generalization of a pre-traine training model to unseen distributions. This offers practical alternative the scenario (unseen combinations) considered in several prior HOI detection papers [19 21.

---

Positional
enoading
Image Encwllrr
Entity text features
Text feature
CE loss
Focal loss
Relation test feutures
Addition
Cross Kncoder
Parallet Emiity Detection
Sequential Relation Inference
Entity "man', "frishee", "cotton sweater"...
Concat
Relation texts: "catching a", "wearing"...
Conditional query generation
In-hatch lahels
Label
Encoder
Entity texts: "strect sign", "wall", Relation texts:" "on top of", "near"..
Out-of-batch labels
sequence
subject queries Q
object queries
Qo
relation queries
Qr
An overview of our pre-training framework, RLIP-ParSe. The Parallel Entity Detection and Sequential Relation Inference blocks represent independent styR- decoders responsible for entity and relation detection, respectively. We omit the localisation loss for clarity. See See. for further details.
Figure 1:
IDN 38 and affordance transfer learning of ATI [20] can be interpreted as entity augmentations to train stronger verb classifier, but these methods do not directly optimise for all the components of HOI detection. Similar to ParSe, CDN (68] explores disentangled embeddings. However, still couples the embedding of subjects and objects, rendering it suboptimal for RLIP. The concurrent work, VLKT [41] aims to derive knowledge from image-level language-image pre-training [52), while we aim to achieve aligned entity- and relation-level pre-training for HOI detection.
Leveraging free-form text for visual pre-training. A series of recent papers have illustrated the significant value of employing free-form language to provide supervision for vision systems. CLIP [52] and ALIGN [25] demonstrated striking improvements in zero-shot image classification ability through contrastive training of image-level representations. Further work has sought to additionally leverage correspondences between objects/regions and text to learn flexible grounding models (2 36 67). We similarly seek to benefit from natural language supervision. However, to the best of our knowledge, we are the first to leverage correspondences between descriptions of relations and explicit pairings of subjects and objects (rather than descriptions of images, objects or regions) as pre-training signal.
3 Methodology
In this section, we first present our triplet detection architecture, ParSe. Second, we describe how ParSe is used to perform relational language- -imagee--mggeeaee mr--trannng (RLIP). Finally, we introduce techniques to synthesise contrastive negatives and mitigate noise and ambiguities among labels. The overall RLIP-ParSe framework is illustrated in Fig.
3.1 ParSe for Triplet Detection
Structure overview. The core idea underpinning the ParSe architecture is to allocate distinct representations of subjects, objects and relations in a holistically optimized model (rather than representing their combination, commonly pursued in prior work (57)). The motivation for doing so two-fold: (i) distinct representations enable the direct use of contrastive RLIP, since these representations can be put in correspondence with separate entity and relation annotations; (ii) the separation of responsibilities allows for a more fine-grained control over the context available for each decision (a theme that has proven important for detection tasks (56)). In particular, note that when detecting subjects and objects, local context is typically most useful. However, when comes to relations, detection will benefit not only from informative local cues, but also neighbouring context (64) (for instance, it useful to be aware of water and hoses when inferring the relation in the triplet <human, wash, car>). To instantiate this idea we follow [68] and implement triplet detection in a two-stage end-to-end manner. Our probabilistic model factorises as follows:
IP(GIQ., Qo, C; Par. Ose) P B, Q., Q., C;0p.r) P(R B., C;0s)
(I)
where Q., Q. E NoDD define two of independent queries for No subjects and No objects: C' denotes features from the detection encoder; B. B., R denote sets of detected subject boxes, object boxes and relations, respectively (these collectively comprise the detection results G): Orar and Sse represent learnable parameters from the entity detection decoder and the relation inferring decoder. respectively. To construct ParSe, design two components, Parallel Entity Detection and Sequential Relation Inference, to implement the second and third terms in Eq. (I) respectively. These described next.
3

---

Parallel Entity Detection. Following the DETR family of architectures B 70 43 we first extract
visual features using an image encoder, add positional encodings and then pass the result through a customized Transformer encoder according to the detector we adopt (we explore both DETR B3] and DDETR (70) variants) to obtain detection features C. Then, two sets of queries Q, and Qo fed into the entity decoder to perform self-attention [58] cross-attention and feed-forward network (FFN) inference, obtaining Q.: Qo E IR No x which are used to predict box locations and classes. Sequential Relation Inference. To encode relations, a sequential step after entity detection (similarly to (68). In the first stage, subjects and objects are detected via Parallel Entity Detection. In the second stage, we adopt a simple parameter- free matching scheme between subjects and objects to generate relation queries: matching by their indices. Using this pairing scheme, we obtain relation queries via a conditional query generation function:
perform Sequential Relation Inference as
Q. = F5o Qo)
(2)
where for simplicity, adopt addition the query generation function. Since we match by indices, Qr E IR Naxh contains Nq relation queries. Qr is then fed into the second decoder to perform Sequential Relation Inference via self-attention, aross-attention and FFN inference to obtain the corresponding relation features Q x which are then used for relation classification.
3.2 RLIP-ParSe for Relational Language-Image Pre-training
For each itcration of pre-training, we construct a minibatch of images and their annotated relation triplets comprising all entities' locations, Ng unique entity text labels as well NR unique relation text labels. We describe how these are used for contrastive pre- pre-training next. Formation of target label sequences. We construct targets from in-batch labels (free-form text descriptions forming subject, object, relation triplets). In more detail, we first aggregate all entity labels within the batch and append to this sequence a no objects label. Next, we similarly aggregate all in-batch relation labels. Then, all entity and relation labels are respectively fed into text encoder (RoBERTa 44 in our implementation) to extract label features denoted as LE and LR. respectively, Note that free-form text label can have multiple tokens after tokenization-we use only the feature derived from the [CLS] token to represent the label. We concatenate the label feature sequence with features from the image encoder shown in Fig. [ To fuse the concatenated features, adopt a simple approach: applying a Transformer encoder (3 28 to obtain fused label features LE E N and L E ND
Cross-modal alignment through classification. To implement RLIP, we task the model with estab- lishing correspondences between entities/relations and their text descriptions using a classification objective, following (36). In particular, we align the ith relation (i) E RD' with relation its text via a Focal loss 42
= Q + (?) + Wa: c,( = (ireal(sigmod((,((1.....
(3)
where (i) + W. is the learnable bias term introduced in 44 W, E RNNxD is a learnable linear projection and W. E N2 is a constant vector filled with -lg (( T) T)/T) with π = .01. The Focal loss is defined via Focal(p) = ((--)) p) log(p) where 7 is set hyperparameter. In the 3 indexes along (i) E R NR. To encourage matching of subjects and objects with their corresponding entity descriptions, an analogous objective to Eq. (3) is used except that softmax and a CE loss are applied and W. is omitted (note that entities are unillabel and relations multi-label, as defined by the downstream task). The central benefit of the RLIP objectives defined above is that they bring the pre-training and downstream HOI detection losses into close alignment since the task of classifying entities and relations in the downstream task reflects the same matching task used in pre-training. As a result, RLIP produces models that can perform HOI detection under zero-shot with no fine-tuning (NF) evaluation protocols.
argument to this loss in Eq.
Label Sequence Extension (LSE). Within a given batch, the number of negative samples available for matching is limited. However, provision of plentiful negatives has been widely shown to improve To this end, we propose Label Sequence Extension as a mechanism to leverage out-of-batch text descriptions. Concretely, we sample additional text descriptions with a ratio of two thirds entity labels and one third relation labels. To ensure computational tractability in the presence of the quadratic complexity of Transformer, we limit the label sequence to predefined length Nr.- We experiment with two sampling strategies: (i) Uniform sampling that draws among candidate labels with equal probability; (ii) Frequency- bassed sampling that samples according to the label frequency in the training set.
contrastive learning 11 23 59).

---

3.3 Addressing Relational Semantic Ambiguity
Datasets with crowd-sourced language annotations [22 33 exhibit significant label noise and ambi- guity. First, the descriptions themselves may be noisy (inaccurate), particularly when the underlying image is challenging to interpret. A second challenge for traditional training schemes is that similar relations can be described differently, thanks to synonyms. For example, the stand near relation may be annotated "stand near", "stand next to", "stand by", etc.. These forms of semantic ambiguity make supervised cross-modal pre-training (which relies on access to consistent labels) challenging. To mitigate this issue, we focus on two aspects of the pre-training input data: (i) the quality of the relation text labels; (ii) the presence of semantically-siminn Relational Quality Labels (RQL). To tackle the first challenge, we propose a label smoothing (46] approach. The key idea is that we expect the difficulty of subject and object detection for a particular instance to correlate with the confidence of the annotated relation. We therefore propose to estimate annotation quality from the quality of the entity detection stage. Drawing inspiration from the
labels in sampled label sequences.
generalised focal loss (37], we instantiate this idea by assessing the quality of the ith subject and object detection after bipartite matching [57] as
e(i) min min(GooU (i, (i))), GIoU (B, B.)()))
(4)
where GloUG-1 denotes generalized IoU from [55] together with a linear scaling function to scale the GIoU value to the range of 0 to 1 and denotes ground-truth annotation. The resulting value
e(i) is then employed calibrate the relation label confidence via multiplication: R(i) = e(i) e(i) Relational Pseudo-Labels (RPL). To address the second issue, we propose a pseudo-labelling strategy
[63 to account for synonyms the extended sequence. We exploit the fact that text embeddings with high semantic similarity will lie close together, as measured by an appropriate distance function (..-). We define the distance between the ith annotated relation label R(i) {0, 1) N' and the jth relation text feature Lıe(i) E RR. from the extended sequence as Nu M (f(()) ŁR()) = Σ k= R(iik) ml (k)), Ri))
(5)
where m(:.)) denotes Euclidean distance. Given the ith relation label, we apply scaling function to (i, via (i.j)) max: (M) M(..) where we have abbreviated M (fi(i), Ri)) (i, for clarity. Next, we use global threshold v) to select label texts with high similarities: we set the jth label in the äth relation labels as (i. j) if M(i, j) > 1j. Note that when applying either RQL or
max, ( (F,A)
RPL, the ground truth labels are continuous (rather than discrete). We therefore employ the Quality Focal Loss 337 (rather than the standard Focal Loss 42) as our objective function.
3.4 Pre-training, Fine-tuning and Inference
By design, our pre-training (RLIP) and fine-tuning phases follow a similar process. For given batch of images with corresponding annotations, we aggregate the results from Parallel Entity Detection and Sequential Relation Inference to form NQ triplets per image. During pre-training and fin-ttnnng, we employ bipartite matching similarly to prior work [5 6 68, 666, following in particular the matching cost proposed in (57). The overall loss is then constructed as follows:
C C= A11 + >gLGIOU + >g(C, + Co) + X,C,
(6) CE loss for
where L11, LOTOU La, La, C, denote the E loss for box regression, GloU loss 155 subject and object classes, and Focal loss for relations (or Quality Focal loss 377 when applying RQL or RPI.. respectively. The X terms are fixed weights to balance multi-task training following [57] with \ 2.5, A 1, A3 1, = 1. During pre-training, the label sequence constructed labels. During fine-tuning, we use all text labels contained in the dataset to form the label sequence (unlike pre-training, these labels fall within a pre-defined category list of limited size). Note that we follow [68] and exclude L. during fine-tuning since HOI detection detects only humans subjects. During inference, the confidence score for object is simply the top-1 score from the softmax distribution over objects, and the relation score is obtained by multiplying the original score from the sigmoid function and the object score. We rank relation scores and filter out the top-K within those correctly localised triplets (IoU > 0.5) for evaluation. K
from both in batbatchhh and out- out-of-batee
is set to 100 by default following (57 68 40 66).
4 Experiments
Datasets. We use the Visual Genome (VG) 32 dataset for RLIP. This dataset contains 108,077 images annotated with free-form text for wide array of objects and relations (100,298 object
5

---

annotations and 36,515 relation annotations). The dataset is pre- processed prior to use (see supple- mentary material for details). For downstream tasks, we conduct experiments on HICO-DET [3] and V-COCO [14]. HICO-DET contains 37.536 training images and 9,515 testing images, annotated with 600 HOI triplets derived from combinations of 117 verbs and 80 objects. We evaluate under the Default setting. V-COCO comprises 2,533 training images, 2,876 validation images and 4,946 testing images annotated with 24 interactions and 80 objects. Results are assessed under two scenarios denoted as AP#11 role and AP#2 role as defined by the official evaluation code 14. Note that all object classes in HICO-DET and V-COCO are identical to COCO.
Implementation details. The basic architecture of the encoder and decoder are based on DETR [3 for ParSe and DDETR [70] for an additional variant named ParSeD. A detailed architecture description of ParSeD (which uses an additional transformer for cross-modal fusion) provided in the supplementary. For Parallel Entity Detection and Sequential Relation Inference, 3 decoding layers used. The number of queries No is set to 100 during pre-training and 64 during fine-tuning (following (88). ? in the Focal loss is set to 2 following [57 68). Nr. in LSE is set to 500 to ensure computational tractability. 17 in RPL is set to 0.3. For pre-training and fine-tuning, the initial learning rate (LR) of the image and text encoders is set to le-5, while all other modules are set to lc-4. For RLIP-ParSeD and ParSeD (object detection and relation detection pre-training), we pre-train model on VG for 50 epochs and drop LR by a factor of 10 at epoch 40. For ParSeD and RLIP-ParSeD, We fine-tune for 60 epochs and drop LR at epoch 40 by a factor of 10. For ParSe and RLIP- fine-tune for 90 epochs and drop LR at epoch 60 by factor of 10. The pre- trainiiinn and fine-tuning strategy follow above descriptions unless stated otherwise. Experiments are conducted on 8 Tesla V100 GPU cards with minibatch size of 32.
We
Experimental protocols. To assess performance under fine-tuning and zero-shot with no fine-tuning (NF) scenarios, we evaluate on HICO-DET across three HOI sets under the mAp metric: Rare (HOls with training samples less than 10, of which there are 138), Kon-Rare (HOIs with samples equal to or more than 10, of which there are 462) and Full (all HOIs, of which there are 600).
To evaluate performance under the zero-shot formulation considered by 20 [9]. report results on unseen combinations (UC). In particular, we report results under two settings: UC with rare-first (UC-RF) selection and UC with non-rare first selection (UC-NF), both of which are assessed across three subsets: Unseen ( (120 HOIs), Seen (480 HOIs) and Full (600 HOIs).
To evaluate few-shot transfer performance, we follow (8 and sample subsets of training annotations from HICO-DET. In detail, sample 1% and 10% of the total annotations available among the HICO-DET training data, ensuring that all objects and verbs (but not all combinations) exist in the
selected annotations. Similarly to the fine-tuning protocol, we evaluate on the Rare, Non-Rart and Full sets.
To assess the robustness of RLIP and its sensitivity to noise in the relation labels, we follow 335 60 and artificially inject noise into the relation labels by randomly flipping a fixed ratio of verbs in HOI triplets across the training set.
4.1 Results and Analysis
Comparing object detection and relation detection pre-training with RLIP. An assessment of the benefits of object detection pre-training using the COCO dataset may offer a somewhat optimistic evaluation of this approach, since COCO shares identical object classes with the downstream HOI detection evaluation datasets HICO-DET and V-COCO (and in the latter case shares training images). If we control for this effect by performing object detection pre-training on VG rather than COCO, we observe a significant drop in performance for our ParSeD baseline (from 29. 12 to 23.78 across the Full set on HICO-DET and from 61.8 to 41.4 on V-COCO for AP#1 rale: as shown in Tab. I. However, since COCO lacks relation annotations, we investigate the benefits of RLIP on VG. We observe that RLIP outperforms vanilla object detection pre-training by wide margin (boosting performance from 23.78 to 29.21 across the Full set on HICO-DET and from 41.4 to 53.1 on V-COCO for AP##1 rote), demonstrating the value of incorporating relations as a pre-training cue. Another way to pre-train with relations is to perform relation detection which is still inferior to RLIP (27.45 < 29.21), demonstrating the importance of relational language-image pre-training.
Leveraging off-the-shelf object detection data without relations. As shown in the previous experiment, while VG provides relation annotations, it provides much weaker basis for object

---

Table 1: Comparisons with previous fine-tuned methods on HICO-DET and V-COCO (column AP
vole and
P ). PT, PTP, OD, RD and MD abhreviate Pre-Training, Pre- Pre-Training Paradigm, Ohject Detection, Relation Detection and Modulated Detection. FRCNN, R. HG and Swin-T denote Faster R-CNN [54), ResNet [7]
Hourglass [ and Swin-Tiny A5]. denotes RLIP is performed on VG, initialized with parameters from
COCO object detection, denotes fine-tuning for 150 epochs following QAHOI 6
PT Data
PTP
Method
Detector
Backbone
Rare Non-Rare Full AP AP# role
role
QAHOI
6
DDETR
Swin-T
22.44
30.27
28.47
ParSeD
DDETR Swin-T
25.76
31.84
30.44
InteractNet 13
FRCNN
R50-FPN
7.16
10.77
9.94
40.0
GPNN S1
FRCNN
R152-DCN
9.34
14.23
13.11
44.0
iCAN 12
FRCNN
ResNet-50
10.45
16.15
14.84
45.3
52.4
UnionDet
29
RetinaNet
R50-FPN
11.72
19.33
17.58
47.5
56.2
PPDM 40
CenterNet
HG104
13.97
24.32
21.94
HOTR 30
DETR
ResNet-50 17.34
27.42
25.10
55.2
64.4
HoTrrasffomer
t
DETR
ResNet-50 16.91
25.41
23.46
52.9
COCO
OD
QPIC [57
DETR
ResNet-50 21.85
31.23
29.07
58.8
61.0
OCN 66
DETR
ResNet-50 25.56
32.51
30.91
64.2
66.3
CDN 68
DETR
ResNet-50 27.39
32.64
31.44
61.7
63.8
QAHOI 6
DDETR
ResNet-50 18.06
28.61
26.18
ParSeD
DDETR
ResNet-50
22.23
31.17
29.12
61.8
64.0
ParSe
DETR
ResNet-50 26.36
33.41
31.79
62.5
64.8
ParSe
DETR
ResNet-101 101 28.59
34.01
32.76
64.4
66.5
OD
ParSeD
DDETR
ResNet-50 19.59
25.03
23.78
41.4
43.0
VG
RD
ParSeD
DDETR
ResNet-50
21.36
29.27
27.45
51.5
53.2
RLIP
RIIP-ParSeD
DDETR
ResNet-50
24.45
30.63
29.21
53.1
55.0
RLIP'
RLIP-PurSeD
DDETR
Re 24.67
32,50
30.70
61.7
63.8
COCO+VG RLIP'
RLIP-PaSSe
DETR
ResNet-50 26.85
34.63
32.84
61.9
64.2
GoldG+
MD
MDETR-ParSe
28
DETR
ResNet-101
22.91
31.07
29.19
53.6
56.0
detection pre- than COCO. More broadly, we may expect that object annotations are likely to be more readily available (and greater in scale) than relation annotations. To mitigate this, a simple solution is to simply load object detection parameters pre-trained from an abject-annotatee (like COCO), to complement the abilities of RLIP. Tab. [ indicates that RLIP-ParSer indeed benefits from this approach, surpassing both object detection pre-training and RLIP (29.12, 29.21 30.70 on the Full set). We pre-trained on DETR for 150 epochs, outperforming an expert object detection pre-training (31.79 + 32.84). On V-COCO, there is a degradation of performance (62.5 61.9) which we believe may be caused by the reduced domain alignment (i.e. common training images) relative to COCO object-detection pre-training [57]
dataset
Comparing cross-modal regional alignment pre-training with RLIP.
We next compare to the
use of language-region alignment pre-training introduced by MDETR 288, which employed the GoldG+ dataset (this comprises VG, COCO and Flickr30k (50 together with the corresponding annotations for referring expressions, VG regions, Flickr entities, and GQA (22). For comparison,
we initialise RLIP-ParSe with MDETR's parameters and then fine-tune on HICO-DET. The results reported in Tab. T as MDETR- ParSe.. Although MDETR makes of heavier backbone (ResNet- 101) and additional pre- pre-traine training data, RLIP-ParS nevertheless surpasses this baseline with lighter ResNet-50 backbone, demonstrating the effectiveness of RLIP for this task.
Zero-shot HOI detection. To assess performance under the zero-shot NF protocol, we compare with other methods using RLIP-ParSe (initialised with COCO parameters followed by RLIP on VG) and ParSe (initialised with COCO parameters). Note that during RLIP, we match subjects against diverse collection of categories. However HOI detection only needs detect person subjects. Consequently, for zero-shot NF inference, we filter out subjects that are not classified as person. We report results in Tab. 2 where we observe that RLIP outperforms several fully fine-tuned methods. We find, as expected, that regional language-imes image pre- training methods like MDETR fail under an NF evaluation, since its pre-training lacks the notion of explicit relations. Under UC-RF and UC-NF protocols (fine-tuning for 40 epochs under UC-NF to avoid over over-fitting). RLIP-ParSe outperforms previous methods and ParSe by performing RLIP on VG. Few-shot transfer on HICO-DET. To
7

---

evaluate few-show transfer, we tune ParSeD for 60 epochs as above, while RLIP-ParSeD is fine-tuned for 10 epochs to avoid over-fitting. The results are shown in Tab. 3 We observe that RLIP significantly benefits few-shot fine-tuning relative to object detection pre-training and relation detection pre-training, especially when data is scarce.
Table 2: Results under zero-shot settings on HICO- Table 3: Few-shot transfer on HICO-DET. OD, RD de- note object detection pre-training and relation detection pre-training.
DET. NR denotes Non-Rare.
Zero-shot
Method
Rare
NR
Full
Method
Data Epochs
Rare Non-Rare Full
MDETR-Puste
28
0.00
0.00 0.00
NF
RLIP-ParSe
15.08
15,50 15.40
ParSeD (VG, OD)
1%
60
0.18
2.05
1.62
10%
60
6.46
12.19
10.87
Zero-shot
Method
Unseen Seen Full
ParSeD (VG, RD)
1%
60
3.74
8.62
7.50
VCL [9
10.06
24.28 21.43
10%
60
12.29
17.98
16.67
ATL 220
9.18
24.67 21.57
UC-RF
FCL 21
13.16
24.23 22.01
ParSeD (COCO, OD)
1%
60
5.86
10.16
9.17
ParSe
18.53
32.21 29.06
10%
60
12.20
20.39
18.51
RLIP-ParSe
19.19
33.35 30.52
0%
12.30
12.81
12.69
VCL 19
16.22
18.52 18.06
RLIP-ParSeD (VG)
1%
10
16.24
16.05
16.09
ATL 20
18.25
18.78 18.67
10%
10
15.43
20.34
19.21
UC-NF
FCL
21
18.66
19.55 19.37
0%
11.20
14.73
13.92
ParSe
19.65
24.50 23.38
1%
10
16.22
18.92
18.30
RLIP-ParSe
20.27
27.67 26.19
(COCO + VG)
10%
10
15.89
23.94
22.09
The influence of relation label noise. To assess sensitivity to noise, we report fine-tuning results on HICO-DET with increasing ratios of relation label noise in Fig. 2 We observe that as label noise increases. the COCO object detection pre-traue vre-trainig adopted by prior work exhibits a greater degradation in performance
-4.60) than RLIP 299.11225.66 -3.53) We also observe that when initialising REIP- with COCO pre-trdined parameters, RLIP again helps to ameliorate noise, with more limited loss of performance (30.70~>26 266.87, -3.83) than COCO pre-training and a similar degradation RLIP-ParSeD with random initialization. Consequently, we deduce that RLIP offers route to mitigating label corruption and improving model robustness [18).
Ablation study of different sampling strategies for label sequence extension using RLIP-ParSeD on HICO-DET.
Table 4:
2/
301
Fine-tuning Sampling Type Rare Non-Rare
Zero-shot (NF)
Full
Rare Non-Rare Full
RLMMSSD IND)
22.58
28.98
27.51
9.77
9.97
9.92
3P%
Uniform
23.33
29.55
28.12
9.46
9.67
9.63
Rekdax Nime Ralo
Frequency- based 23.02
29.77
28.22
10.45
11.26
11.07
Figure 2: Relation Label Noise.
4.2 Ablation studies and analysis
Ablation study of ParSe on the influence of decoupled representations. We report ablation study of the ParSe architecture in Tab. 5o highlight the importance of decoupling the representation of subjects, objects and relations. The first row of Tab. representt the use of coupled representations for subjects, objects and relations [57]. The second row of Tab. 5 represents the use of coupled representations for subjects and objects that are disentangled from relations [68). The final row (ParSe) uses fully-disentangled representations. We observe a clear gain resulting from ParSe over methods using joint representation of (some subset of) subject, object and relation triplets.
Ablation study of sampling strategy. In Tab. 4 we present an ablation study to assess the efficacy of out-of-batch sampling strategies. Intuitively, uniform sampling will up-weight descriptions from the tail the distribution while frequency-bur improvements to fine-tuning by providing additional negative samples. However, by over-sampling descriptions from the tail, uniform sampling performs less well with common texts in the downstream task, and thus fares less well overall.
sampling will preserve the distribution. Both bring
Visualisation of ParSe attention weights. We visualise attention weights from ParSe for several example images in Fig. 3 We observe that ParSe attends distinct regions for subjects, objects and relations. This aligns with our motivation that entity detection is best supported by local context,
8

---

Table 5: Fine-tuning results with ParSe (COCO, OD) on HICO-DET.
ParSe Architecture
Coupling
Rare
Non-Rare
Full
coupled subject, objects and relations
23.18
31.45
29.55
w/ Se
coupled subject and objects fully decoupled
25.58
32.50
30.91
w/ ParSe
26.36
33.41
31.79
Table 6: Ablation study of RLIP techniques using RLIP-ParSeD on HICO-DET.
RLIP Technique
Fine-tuning Non-Rare
Zero-shot (NF)
Relation
LSE
ROL RPL
Rare
Full
Rare
Non-Rare
Full
Uniformity.
Alignment]
22.58
28.98
27.51
9.77
9.97
9.92
-0.8233
0.3650
23.02
29.77
28.22
10.45
11.26
11.07
-1.0556
0.4542
24.32
30.32
28.94
11.49
12.60
12.34
-1,3986
0.6072
24.45
30.63
29.21
12.30
12.81
12.69
-1.3265
0.5799
while relation inference draws on additional spatial context, conditioning on subjects and objects like hands and string, as well as the wet ground and sky where appropriate.
Ablation study of RLIP techniques. In Tab. we present an ablation study of the three proposed technical contributions. We observe that each benefits both fine-tuning and zero-shot NF under all metrics. LSE attains a greater boost for the Non-Rare set (by sampling according to the training set distribution). On the other hand, RPL enhances results for the Rare set, likely due to the propensity of RPL to label rare descriptions as positive (due to the long-taile tailed distribution of text labels).
human
hold
backpack
human
wash
bus
Understanding LSE, ROL and RPL with Uniformity and Alignment. To gain insight into representation quality, Wang al. (59) proposed two metrics, uniformity and alignment, which we employ here to better understand the influence of our contributions. To this end, perform zero-shot (NF) evaluation on HICO-DET, using bipartite matching to assign predicted triplets to ground-truth labels. We then calculate uniformity and alignment metrics for relation fea- [e- (e-(l]((e)-/vlllli) and
human
fly
kite
Figure Attention weight analysis for the top-1 scored verb. Weights are extracted from Parallel Entity Detec- tion (human and object) and Sequential Relation Inference (verb).
tures via (u = log(E ( A Pasta
(x.9)
La (f; F(e.,))~PPP.. (x)) [1)) where a, = 2. We present the results in Tab. 6 where observe that LSE and RQL both reduce uniformity, the former through additional negative pairs and the latter by reducing the loss assigned to over-confident text labels. We also observe that RPL yields better alignment through discovering additional positive labels. The results indicate, however, a trade-off between the metrics a useful direction for future work would be determine how to find an appropriate balance between them.
Verb-wise mAP Analysis for zero-shot (NF) evaluation. We provide analysis to give sense of the verb overlap of HICO with VG. We use "relationship aliases" from the official VG website obtain many HOI verb annotations from VG as possible by string matching. The result is shown in Tab. in the supplementary material. We observe that in VG there are only 2,203 HOI verb annotations even when considering relationship aliases-approiimant approximately 1.47% of the number of relationship annotations in HICO-DET. 30 HOI verbs do not have an annotation and 45 HOI verbs have five or fewer annotations. In RLIP- ParSe (COCO+VG), verbs is 5.56 while mAP for the remaining 87 verbs 18.12. If we use uni-modal relation detection pre-training, the result for the 30 verbs degrades to zero. In light of this, we conjecture that existing relations can transfer their knowledge to the inference of non-existing relations in HOI detection. To provide a more detailed analysis, we show the verb-wise mAP on HICO verbs in VG (Fig. 3) and not in VG (Fig. 4) with zero-shot (NF) evaluation (figures are provided in the supplementary material), where we observe solid performance for some verbs.
observe that mAP for the 30
Probing into reasons for the verb zero-shot performance. We aim to qualitatively understand where the zero-shot ability stems from. In the above analysis, pay has the highest performance among verbs not seen by VG (Fig. 4 in the supplementary material). In the methodology section,

---

Table 7: VG verb ranking given similar subject-object triplets from HICO-DET. Verbs are in ascending order of Euclidean distance. (The Cosine distance can also output similar rankings.)
"pay"
("parking meter") putting money in collecting money at puts change into repairing checking next to
Count
1
1
Fuclidean 0.4560
11,56
11.70
13.34
14,21
15.16
16.12
Cosine
0.4576
0.3
0.2554
0.1583
0.0709
present the conditional query generation that constrains the verb inference to be related to subjects and objects, providing verb inference with conditional context. Thus, to analyze how this ability of verb zero-shot inference emerges, we need to consider the subject and object context as they are essential to predict the verb in ParSe. For the verb pay in HICO-DET, there is only one possible triplet annotated, "person pay parking meter". Then, we want to answer, "Is there any triplet annotated with similar or identical subjects and objects that transfer the inference ability to pay?" To answer this question, we search for triplets annotated with similar subjects and objects to HICO-DET from VG (For details, please refer to the analysis of Tab. 12 in the supplementary material.). We report the verb distribution of the limited number of triplets that are found, ranking the verbs in ascending order of Euclidean distance to the target verb (Tab. 7 From this table, we can see that the verbs quantitatively closer (in Euclidean distance or Cosine distance) to pay have similar meanings to pay, shown by their lexical variants or grammatical variants (e.g.. putting money in has a similar meaning to pay). Thus, in the VG dataset, there is human putting money in parking meter, which may transfer to the zero-shot recognition of person pay parking meter in HICO-DET. More examples can be found in Tab. 12 in the supplementary material. In short conclusion, we conjecture that the zero-shot inference ability of RLIP is not from the scale of annotations (by comparing relation detection pre-training and RLIP using VG), but the ability to transfer the verb inference knowledge from semantically similar annotations. This analysis also accords with previous works [52 36 that semantic diversity is important as it introduces large-scale potential annotations, ensuring a model transfers well to different data distributions.
Second, we aim to demonstrate quantitatively how RLIP pre-trains the model to perform zero- shot detection from the perspective of represen- tation learning. We employ the Uniformity met- ric introduced in [59]. Uniformity is a metric to assess a model's 's generalization in contrastive learning. In this case, since label textual embed- dings serve as a classifier in RLIP, we calculate the Uniformity of the seen verbs, unseen verbs and all verbs, aiming observe how the generalization changes before and after RLIP, and how the generalization varies between seen verbs and unseen verbs. The results are shown in Tab. 8 As can be seen from the table, Uniformity values are high before RLIP, suggesting that the representations before RLIP are distributed compactly, leading to poor classifier. However, after RLIP is performed, the 87 seen verbs have a substantially lower Uniformity value, corresponding with decent zero-shot performance. Similarly, the 30 unseen verbs and the combination of 117 verbs also have excellent Uniformity values, contributing to unseen zero-shot performance. Through this quantitative observation, we think that from the perspective of representations, RLIP contributes to improved zero-shot performance.
Table 8:
Uniformity analysis of the seen verbs, unseen verbs and all verbs before and after RLIP. Lower uniformity value is better.
Verb Set
Seen (87) Unseen (30) All (117)
Before RLIPP
-0.00367
-0.00436
-0.00388
After RLIP
-3.73780
-3,59457
-3.71330
From all the above analysis, we conjecture that the zero-shot performance is not caused by the increased dataset size or annotations, but rather from the generalization in representations obtained by pre- preitrain with language supervision.
5 Conclusion
In this paper, we propose RLIP as a pre-training strategy for HOI detection. We show that RLIP, together with our additional technical contributions, boosts HOI detection performance under fine- tuning, zero-shot and few-shot evaluations, and improves robustness against noisy annotations.
Acknowledgemeetss: We would like to appreciate anonymous reviewers for their valuable feedback and members from Fundamental Vision Intelligence Team of Alibaba DAMO Academy for sharing computational resources. This work was supported in part by National Natural Science Foundation of China Grant No. 62173298 and by Alibaba Group through Alibaba Research Intern Program.
10

---

References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian. Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34, 2021.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198, 2022.
[3] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. arXiv preprint UrXiv:2005.1287, 2020.
[4] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human- object interactions. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 381-389. IEFE, 2018.
[5] Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. Hico: benchmark for recognizing human-object interactions in images. In Proceedings of the IEEE International Conference on Computer Vision, pages 1017-1025, 2015.
[6] Junwen Chen and Keiji Yanai. Qahoi: Query-based anchors for human-object interaction detection. arXiv preprint arXiv:2112.08647, 2021.
[7] Mingfei Chen, Yue Liao, Si Liu, Zhiyuan Chen, Fei Wang, and Chen Qian. Reformulating hoi detection as adaptive set prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9004-99 2013, 2021.
[8] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pages 1597-1607. PMLR, 2020.
[9] Yuren Cong, Michael Ying Yang, and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation. arXiv preprint arXiv:2201. /460, 2022.
[10] Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 7373-7382, 2021.
[11] Yutong Feng, Jianwen Jiang, Mingqian Tang, Rong Jin, and Yue Gao. Rethinking supervised pre-traiming for better downstream transferring. In International Conference on Learning Representations, 2022.
[12] Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance-centric attention network for human-object interaction detection. arXiv preprint arXiv: 10437, 2018.
[13] Georgia Gkioxari, Ross Girshick, Piotr Dollár, and Kaiming He. Detecting and recognizing hujan-object interactions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8359-8367, 2018
[14] Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint 1055041505.04474, 2015.
[15] Tanmay Gupta, Alexander Schwing, and Derek Hoiem. No-frills human-object interaction detection: Factorization, layout encodings, and training techniques. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9677-9685, 2019.
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. Proceedings of the IEEE International Conference on Computer Vision, pages 2961-2969, 2017.
[17] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.

---

[18] Dan Hendrycks, Kimin Lee, and Mantas Mazeika. Using pre- traininn can improve model robustness and uncertainty. In International Conference on Machine Learning, pages 2712- 2721. PMLR, 2019.
[19] Zhi Hou, Xiaojiang Peng, Yu Qiao, and Dacheng Tao. Visual compositional learning for human- object interaction detection. In European Conference Computer Vision, pages 584 600. Springer, 2020.
[20] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Affordance transfer learning for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 195-504, 2021.
[21] Zhi Hou, Baosheng Yu, Yu Qiao, Xiaojiang Peng, and Dacheng Tao. Detecting human-object interaction via fabricated compositional learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14646-14655 2021.
[22] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5700-6709, 2019.
[23] Ashraful Islam, Chun-Fu Chen, Rameswar Panda, Leonid Karlinsky, Richard J. Radke, and Rogério Schmidt Feris. broad study on the transferability of visual representations with contrastive learning. 2021 IEEE/CVF International Conference on Computer Vision, pages 8825-8835, 2021.
[24] Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10236-10247, 2020.
[25] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V. Le, Yun- Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[26] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1219-1228, 2018.
[27] Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3668-3678, 2015.
[28] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulee of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
detection for end-to-end multi-modal understanding. In Proceedings
[29] Bumsoo Kim, Tacho Choi, Jaewoo Kang, and Hyunwoo J Kim. Uniondet: Union-level detector towards real-time human-objec interaction detection. In European Conference on Computer Vision, pages 498-514. Springer, 2020.
[30] Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Fun-Sol Kim, and Hyunwoo! J Kim. Hotr: End-to- end oujenn-obect interaction detection with transformers. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 74 74-83, 2021.
[31] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. Detecting human-object interactions with action co-occurrence priors. In European Conference on Computer Vision, pages 718-736. Springer, 2020.
[32] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision, 123(1):32-77) 2017.
[33] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov. Mattco Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 1956-199999991.. 2020.
12

---

[34] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare. Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Aligm before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34, 2021.
[35] Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- semi-supervised learning. In International Conference on Learning Representations, 2020.
[36] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jeng-Neng Hwang, et al. Grounded language-image pre-training. arXiv preprint arXiv:2112.038577 2021.
[37] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang. Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection. Advances in Neural Information Processing Systems, 33:21002-21012, 2020.
[38] Yong- Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. Advances in Neural Information Processing Systems, 33, 2020.
[39] Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for cumannobjeet interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3585-3594, 2019.
[40] Yue Liao, Si Liu, Fei Wang, Yanjie Chen, Chen Qian, and Jiashi Feng. Ppdm: Parallel point detection and matching for real-time human-object interaction detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 482-490, 2020.
[41] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vikt: Simplify association and enhance interaction understanding for hoi detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20123- 20132, June 2022.
[42] Tsung-Yr Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE International Conference Computer Vision, pages 2980-2900999 2988, 2017.
[43] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201. 12329, 2022.
[44] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach. arXiv preprint arXiv: /907. 1692, 2019.
[45] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[46] Michal Lukasik, Srinadh Bhojanapalli, Aditya Menon, and Sanjiv Kumar. Does label smoothing mitigate label noise? In International Conference on Machine Learning, pages 6448-6458. PMLR, 2020.
[47] Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European Conference on Computer Vision, pages 489.499. Springer, 2016.
[48] Boxiao Pan, Haoye Cai, De-An Huang, Kuan-Hui Lee, Adrien Gaidon, Ehsan Adeli, and Juan Carlos Niebles. Spatio-temporal graph for video captioning with knowledge distillation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10870-10879, 2020.
[49] Julia Peyre, Ivan Laptev, Cordelia Schmid, and Josef Sivie. Detecting unseen visual relations using analogies. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1981-1990, 2019.
13

---

[50] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to- to-phrase correspondences for richer image-to-sentenge models. In Proceedings of the IEEE International Conference on Computer Vision, pages 2641-2649, 2015.
[51] Siyuan Qi, Wenguan Wang. Baoxiong Jia, Jianbing Shen, and Song- Chun Zhu. Learning human-object interactions by graph parsing neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal. Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. arXiv preprint arXiv:2103. 00020, 2021.
[53] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
[54] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
[55] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, lan Reid, and Silvio Savarese. Generalized intersection over union: A metric and loss for bounding box regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 658-666, 2019.
[56] Guanglu Song, Yu Liu, and Xiaogang Wang. Revisiting the sibling head in object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11563-11572. 2020.
[57] Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpie: Qusey-based pairwise human- object interaction detection with image-wide contextual information. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10410-10419, 2021.
[58) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- fion Processing Systems, pages 5998 6008, 2017.
[59] Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In International Conference on Machine Learning, pages 9929-9939. PMLR, 2020.
[60] Zhi-Fan Wu, Tong Wei, Jianwen Jiang, Chaojie Mao, Mingqian Tang. and Yu-Feng Li. Nge: unified framework for learning with open-world noisy data. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 62-71, October 2021.
[61] Bingjie Xu, Yongkang Wong, Junnan Li, Qi Zhao, and Mohan Kankanhalli. Learning to detect human-object interactions with knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019.
[62] Ting Yao, Yingwei Pan, Yehao Li, and Tao Mei. Exploring visual relationship for image captioning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 684 699,,99 2018.
[63] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd annual meeting of the association for computational linguistics, pages 189-196, 1995.
[64] Hangjie Yuan and Dong Ni. Learning visual context for group activity recognition. In Pro- ceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 3261-3269, 2021.
[65] Hangjie Yuan, Dong Ni, and Mang Wang. Spatio-temporal dynamic inference network for group activity recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7476-7485, 2021.
14

---

[66] Hangjie Yuan, Mang Wang, Dong Ni, and Liangpeng Xu. Detecting human human-object interactions with object-guided cross-modal calibrated semantics. Proceedings of the AAAI Conference on Artificial Intelligence, 36(3):3206-321 3214, Jun. 2022.
[67] Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08276. 2021.
[68] Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li. Mining the benefits two-stage and one-stage hoi detection. Advances in Neural Information Processing Systems, 34, 2021.
[69] Xubin Zhong, Xian Qu, Changxing Ding, and Dacheng Tao. Glance and gaze: Inferring action- aware points for one-stage human-object interaction detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13234-13245 2021.
[70] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
[71] Cheng Zou, Bohan Wang. Yue Hu, Junqi Liu, Qian Wu, Yu Zhao, Boxun Li, Chenguang Zhang, Chi Zhang, Yichen Wei, al. End-to-end human object interaction detection with hoi transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11825-11834, 2021.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] It's mentioned in paragraph "Leveraging off-the-shelf object detection data without relations", and discussed in more detail in the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] It's included the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material
URL)? [Yes] Data is publicly available and instructions are included in the experiment section.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] They are included in Methodology and Experiments section.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes] It's included in the supplementary
material.
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [No]
15

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "vBW5KiYxzHd", "reviewer": "Reviewer_4wiK", "summary": "This paper aims to adopt the language-image contrastive pre-training techniques to boost the performance and robustness of the Human-object Interaction (HOI) detection task. For this purpose, this paper first modifies the conventional DETR-based HOI detection framework by decoupling detection and interaction classification and disentangling the subject and object queries. Then, this paper converts the entity labels and relation labels into text and embeds them into a latent space, and constructs contrastive pairs in or out of batches. Extensive experiments have been conducted in HICO-Det and V-COCO datasets from regular, few-shot, and zero-shot settings.", "questions": "- My main concerns and questions lie in the 'Missing important baseline' in the weaknesses. The author should provide the results with the mentioned setting for a fair comparison to verify the effectiveness of the proposed RLIP.\n\n- Confused by the OD+VG performances. VG is a large-scale visual relationship detection dataset, but the performance has dropped a lot when pre-training in this dataset. A good choice may be to train DETR in VG first and then train in COCO because the object labels of HICO-Det are the same as COCO. A detailed analysis of VG should be included.", "limitations": "The authors provided limitations in supplementary materials.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea and motivation of this paper are generally reasonable.\n- The proposed method can be easy to adapt zero-shot, few-shot, and regular HOI detection tasks.\n- The writing of this paper is generally clear and the proposed method is simple and easy to follow. \n\nWeaknesses\n- The proposed architecture is not new in the HOI detection area, which combines the CDN [1] and PST [2], and the idea is entirely the same as the proposed architecture GEN in the recent paper GEN-VLKT [3]. Though GEN-VLKT is a CVPR 2022 paper, it has been uploaded in Arxiv and released code in Mar. 2022. Thus the author may present a detailed discussion with this paper.\n\n- Limited Application Scenarios. The proposed pre-training techniques still require relation triplets annotations, which are expensive to obtain.\n\n- Missing details about OD+VG. The DETR requires a long time to converge, so the author should report the number of epochs to pre-train the DETR in VG. The significant performance gap between OD and RLIP with VG may come from the insufficient pre-training for OD+VG.\n\n- Missing important baseline. The OD+VG baseline only trains a detector in VG with the detection annotation, but RLIP adopts the relation and detection annotations in the VG simultaneously. Thus a fair baseline is to directly to pre-train the ParSe with the VG visual relationship detection based on the COCO pre-trained model.\n\n- An additional large-scale dataset with a minor improvement. Equipped with COCO+VG pre-training, the performance only improved by 1.05 mAP (31.79->32.84).\n\n[1] Mining the Benefits of Two-stage and One-stage HOI Detection. NIPS 2021.\n[2] Visual Relationship Detection Using Part-and-Sum Transformers with Composite Queries. ICCV 2021.\n[3] GEN-VLKT: Simplify Association and Enhance Interaction Understanding for HOI Detection. Arxiv Mar. 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "r0X9ieSkI3G", "reviewer": "Reviewer_MVep", "summary": "This work proposes a visual-language pre-training method for HOI detection. Following MDETR and CDN, a pipeline named ParSe is proposed to implement contrastive learning upon humans, objects, and relations. Correspondingly, a pos-neg language labels scheme and a language ambiguity suppression method are proposed for the pre-training. On commonly-used benchmarks, the proposed method is compared with SOTAs on both supervised and zero-shot settings.", "questions": "1. Efficiency of the sequential h-o pairing? \n\n2. How similar of the pos and neg prompts? Need an analysis.\n\n3. As the pre-training using VG, many zero-shot relations of HICO-DET maybe not zero-shot no more, as there may be many similar and even same relations in VG for training. So a detailed analysis should be taken to probe the **real \"zero-shotness\"**.\n\n4. In Eq. 5, is the Euclidean distance the best choice? How about the others like cosine distance?", "limitations": "N/A.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n+ The HOI contrastive learning is an interesting topic and would advance the HOI detection, considering the possibility of open-vocabulary visual-language learning.\n\n+ Some reasonable model designs are proposed to handle the pos-neg label scheme, language label processing, etc.\n\n+ Extensive ablation studies are conducted and presented. Some analysis is useful for follow-ups.\n\n+ Borrowing the idea from MDETR into the v-l HOI contrastive learning is non-trivial.\n\nCons:\n- First, is the novelty of the first contribution. As mentioned by the authors too, they follow CLIP, CDN, and MDETR to build the ParSe model. Though under the v-l setting, few novel designs are proposed to bring new insights. \n\n- Second, the experiment. In Tab 1, the comparison between methods is mainly within the setting using Res-50 which is reasonable, but for CDN, some results are missing: CDN-B with Res-50: 27.55 33.05 31.78 (which is comparable with the proposed method 26.36 33.41 31.79 in COCO as PT data); and many previous works are also not compared here. If compared with the above result of CDN, it seems marginal improvement is achieved. Though performance does not mean all, considering the similar structure of the proposed method with CDN, concern raises. Meanwhile, if using Res-101 and Swin-tiny, how would ParSe perform, and does it have an advantage compared with CDN and QAHOI as the same series methods. Moreover, ParSe can use extra data in the pre-training, but few improvements are achieved. \n\n- In the discussion of the first contribution, in fact, many previous methods (iCAN, TIN, etc) adopted the separated representations for a human, object, and relations. Even using a transformer, the differences and similarities should also be discussed. And CDN is another case, which is followed by this work. Thus, I suggest revising the part about the first contribution.\nAs for the relation contrastive learning, there are some recent works like Contrastive Visual and Language Translational Embeddings for Visual Relationship Detection, Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationships (just for your information, does need to discuss as they are too new).\nL84: open-vocabulary recognition remains underexplored. This point is also open to discussion, as VCL, FCL,  Detecting Unseen Visual Relations Using Analogies, etc. also have considered open h-v-o scenarios.\nL88: rendering it suboptimal for RLIP. Why, please give a more detailed discussion.\n\n- Method part is somewhat hard to follow as the dense information presented is a short section. And many adopted methods are just given a citation without a brief introduction, which makes the reading interrupted.\n\n- Prompt diversity needs a detailed analysis. \n\n- Is the zero-shot setting fair (tab 2) as the proposed method can use extra data and the other method do not? Please clarify this.\n\nTypo: L63 Parse --> ParSe", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "hL0EmVH3E0", "reviewer": "Reviewer_VZWS", "summary": "-- The paper proposes Relational Language-Image Pre-training(RLIP) a pre-training paradigm for Human-Object Interaction (HOI) Detection. This paper posits that this pre-training methodology aims to overcome the gaps left by Object Detection pre-training methodologies that a not fully tailored towards HoI Detection. \n\n-- The paper presents multiple modules to achieve this pre-training objective. Parallel entity detection and Sequential relation inference (ParSe) proposes a DETR-like training architecture that employs query groups for Subject and Object representation and further query groups conditional on these for modeling Relational representations. To enable contrastive learning during training, the paper proposes  Label Sequence Extension (LSE) that performs out-of-batch-sampling to improve quality of negative samples. Finally, to address label noise and ambiguity in relationship modeling, the paper proposes employing Relational Quality Labels (RQL) and Relational Pseudo-Labels (RPL) modules during training.\n\n-- The paper reports the benefits of the pre-training methodology for HOI detection methods for fine-tuned, zero-shot and few-shot settings.", "questions": "-- Did the authors try any dataset other than VG with similar entity description to showcase the robustness of the model?\n\n-- The authors could try to include more qualitative examples in the supplementary material to showcase success and failure cases.\n\n-- There are couple of small changes that could be made in writing\n\n    - #L57 'negatives samples'  -> 'negative samples'\n    - #L144 'We next similarly' -> Next, we similarly", "limitations": "-- There are no suggestions on the societal impact front --", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper leverages the entity descriptions of Visual Genome (VG) datasets efficiently to improve the performance on HOI detection, one of the challenging but relevant problems for scene understanding.  The proposed ParSE method leverages the relational entities with context along with subject and object entities to improve <subject-object-relation> prediction.\n\n+ The paper proposed modules to refine the label noise and semantic ambiguity in the VG dataset using the RQL and RPL modules which is a relevant problem for VG dataset.\n\n\n+ The paper presents strong results for the zero-shot HOI detection over other SoTA approaches demonstrating the effectiveness of pre-training method for zero-shot detection. The paper also shows the contribution of each of the modules through ablation studies showcasing the contributions of each of the modules.\n\n- The proposed method does not show a strong improvement over the CDN method for fully-supervised HOI detection method for both VCOC and HICO-DET datasets. Infact, the performance of the method goes down when including COCO dataset for RLIP compared to just the OD pre-training method using ParSE indicating overfitting of the pre-training method when including COCO.\n\n- The improvement for zero-shot setting could be down to the strong prior in the distribution provided by the VG dataset since the zero-shot formulation is based on OOD rather Unseen setting. Including another dataset with triplet entities of Subject, Object, Relation might showcase the robustness of the method to different data distributions.\n\n- The method ranks low in novelty since it reuses many of the existing HOI detection methodologies and adapts the same for pre-training using VG dataset.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "LAzp5br9CW8", "reviewer": "Reviewer_7c46", "summary": "For Human-Object Interaction (HOI) detection, the authors propose Relational Language-Image Pre-training (RLIP), a strategy for contrastive pre-training that leverages both entity and relation descriptions. \nTo make effective use of such pre-training, they make three technical contributions: \n(1) a new Parallel entity detection and Sequential relation inference (ParSe) architecture that enables the use of both entity and relation descriptions during pre-training; \n(2) a synthetic data generation framework, Label Sequence Extension, that expands the scale of language data available within each minibatch; \n(3) ambiguity-suppression mechanisms, Relation Quality Labels, and Relation Pseudo-Labels, to mitigate the influence of ambiguous/noisy samples in the pre-training data. \nThrough extensive experiments, they demonstrate the benefits of these contributions, collectively termed RLIP-ParSe, for improved zero-shot, few-shot, and fine-tuning HOI detection performance as well as increased robustness to learning from noisy annotations.\n", "questions": "In Table 1, even without the RLIP paradigm, how do ParSe and ParseD show favorable performance compared to the previous method?", "limitations": "The authors addressed the limitations of the proposed method and the potential negative social impact of their work.", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n\n- The authors show the effectiveness of the proposed method on various setups such as few-shot and zero-shot setups.\n\n- The proposed ParSe model consistently shows favorable performance compared to the existing methods.\n\nWeakness\n\n- The original contribution of the ParSe model is unclear. In Sec.3, the authors should further emphasize the technical significance of the proposed ParSe model compared to the existing architectures. From the current explanation, it is hard to see the novelty of the proposed model.\n\n- In Table 1, in order to more comprehensively validate the effectiveness of the proposed pre-training method with external data, it would be better to compare with the existing methods when using VG data when using VG data as well.\n\n\n========== ------- Comments after the rebuttal ------======== \n\nI have read the other reviewer's comments and the author's rebuttal, which addresses most of my concerns. Therefore, I would like to raise my score.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept