Weakly-supervised HOI Detection via Prior-guided Bi-level Representation Learning
Authors: Bo Wan
Yongfei Liu
Desen Zhou
Tinne Tuytelaars
Xuming He
Venue: ICLR 2023 poster (2023)
Keywords: HOI Detection
Weakly-supervised Learning
CLIP-guided Representation Learning

Abstract: Human object interaction (HOI) detection plays a crucial role in human-centric scene understanding and serves as a fundamental building block for many vision tasks. One generalizable and scalable strategy for HOI detection is to use weak supervision, learning from image-level annotations only. This is inherently challenging due to ambiguous human-object associations, large search space of detecting HOIs and highly noisy training signal. A promising strategy to address those challenges is to exploit knowledge from large-scale pretrained models (e.g., CLIP), but a direct knowledge distillation strategy does not perform well on the weakly-supervised setting. In contrast, we develop a CLIP-guided HOI representation capable of incorporating the prior knowledge at both image level and HOI instance level, and adopt a self-taught mechanism to prune incorrect human-object associations. Experimental results on HICO-DET and V-COCO
show that our method outperforms the previous works by a sizable margin, showing the efficacy of our HOI representation.

Published as a conference paper at ICLR 2023
WEAKLY-SUPERVISED HOI DETECTION VIA PRIOR- GUIDED BI-LEVEL REPRESENTATION LEARNING
Bo Wan : Yongfei Liu 2- Desen Zhou 2, Tinne Tuytelaars Xuming He
KU Leuven, Leuven, Belgium: 2 ShanghaiTTcch University. Shanghai, China
3 Shanghai Engineering Research Center of Intelligent Vision and Imaging Ibwan, tinne. tuytelaars @esat kuleuven.be [liuyf3, zhouds, hexm) Dshanghaitech edu.cn
ABSTRACT
Human object interaction (HOI) detection plays a crucial role in human-centric scene understanding and serves as fundamental building-block for many vi- sion tasks. One generalizable and scalable strategy for HOI detection is to use weak supervision, learning from image-level annotations only. This is inherently challenging due to ambiguous human-object associations. large search space of detecting HOIs and highly noisy training signal. A promising strategy to address those challenges is to exploit knowledge from large-scale pretrained models (e.g-, CLIP), but a direct knowledge distillation strategy (Liao et al., (202)) does not per- form well on the weakly-superviee supervised setting. In contrast, we develop CLIP-guided HOI representation capable of incorporating the prior knowledge at both image level and HOI instance level, and adopt self-taught mechanism prune incorrect hujan-object associations. Experimental results on HICO-DET and V-COCO show that our method outperforms the previous works by a sizable margin, showing the efficacy of our HOI representation.
INTRODUCTION
Human object interaction detection aims to simultancously localize the human-object regions in an image and to classify their interactions, which serves as a fundamental building-block in a wide range of tasks human-centric artificial intelligence, such as human activity recognition (Heilbron et al..
2015; Tina et al. 2021), human motion tracking (Wafae et al. [0199] Nishimura et al. 2021 ) and
anomalous behavior detection (Liu et al. 2018 Pang et | 2020).
Usually, HOI detection adopts supervised learning paradigm (Gupta & Malik 2015 Chao et al. 2018. Wan et al., 2019 Gao et al. 2020 Zhang al., 2021c). This requires detailed annotations (i.e. human and object bounding boxes and their interaction types) in the training stage. However, such HOI annotations are expensive to collect and prone to labeling errors. In contrast, it is much casier to acquire image-level descriptions of target scenes. Consequently, more scalable strategy for HOI detection is to learn from weak annotations at the image level, known as weakly-supervisee HOI detection (Zhang et al. 2017). Learning under such weak supervision is particularly challenging mainly due to the lack of accurate visual-semantic associations, large search space of detecting HOIs and highly noisy training signal from only image level supervision.
Most existing works (Zhang et al. 2017 Baldassarre et al. 2020 Kumaraswamy et al. 2021) attempt tackle the weakki--upervviee HOT detection in Multiple Instance Learning (MIL.) framework (Ilse) et al. 2018). They first utilize an object detector to generate human-object proposals and then train an interaction classifier with image-level labels as supervision. Despite promising results, these methods suffer from several weaknesses when coping with diverse and fine-grainet HOIs. Firstly, they usually rely on visual representations derived from the external object detector, which mainly focus on the semantic concepts of the objects in the scene and hence are insufficient for capturing the concept of fine-grained interactions. Secondly, as the image-level supervision tends to ignore the imbalance in HOI classes, their representation learning is more susceptible to the dataset bias and dominated by frequent interaction classes. Finally, these methods learn the HOI concepts from a candidate set generated by pairing up all the human and object proposals, which is highly noisy and often leads to erroneous human-object associations for many interaction classes.
"Equal Contribution. Code is available at http:://ghhab.ccm//obbw/a/////// wwtp:///wwwww///99////////

---

Published as a conference paper at ICLR 2023
To address the aforementioned limitations, introduce strategy. It aims to incorporate the prior knowledge from pretrained foundation models to facilitate the HOI learning. In particular, we propose to integrate CLIP (Radford et al. 2021b). a large-scale Iasion-language pretrained model. This allows us to exploit the strong generalization capability of the CLIP representation for learning a better HOI representation under weak supervision. Compared to the representations learned by the object detector, the CLIP representations are inherently less object-centrie, hence more likely to incorporate also aspects about the human-object interaction, as evidenced by Appendix[A] Although a few works have successfully exploited CLIP for supervised HOI detection in the past, experimentally we find they do not perform well in the more challenging weakly-supervised setting (c.f. Appendix B). We hypothesize this is because they only transfer knowledge at image level, and fail without supervision at the level of human-object pairs.
weakly-supervisee HOI detection
To this end, we develop CLIP-guided HOI representation capable of incorporating the prior knowledge of HOIs at two different levels. First, at the image level, we utilize the visual and linguistic embeddings of the CLIP model to build global HOI knowledge bank and generate image-level HOI predictions. In addition, for each human-object pair, we enrich the region-based HOI features by the HOI representations in the knowledge bank via novel attention mechanism. Such bi-level framework enables us to exploit the image-level supervision more effectively through the shared HOI knowledge bank, and to enhance the interaction feature learning by introducing the visual and text representations of the CLIP model.
We instantiate our bi-level knowledge integration strategy modular deep neural network with a global and local branch. Given the human-object proposals generated by an off-the-shelf object detector, the global branch starts with backbone network to compute image feature maps, which are used by a subsequent H101 recognition network to predict the image-wise HOI scores. The local branch builds a knowledge transfer network to extract the auman-object features and augment them with the CLIP- L.lidguieed knowledge bank, followed by a pairwise classification network to compute their The relatedness scores are used to prune incorrect human-object associations, which mitigates the issue of noisy proposals. Finally, the outputs of the two branches are fused to generate the final HOI scores.
relatedness and interaction scores |]
To train our HOI detection network with image-level annotations, we first initialize the backbone network and the HOI knowledge bank from the CLIP encoders, and then train the entire model in an end-to-end manner. In particular, we devise novel multi-task weak supervision loss consisting of three terms: 1) an image-level HOI classification loss for the global branch; 2) an MIL-like loss for the interaction scores predicted by the local branch, which is defined on the aggregate of all the human-object pair predictions; 3) a self-taught classification loss for the relatedness of each human-object pair, which uses the interaction scores from the model itself as supervision.
We validate our methods on two public benchmarks: HICO-DET (Chao et al.] 2018) and V- COCO (Gupta & Malik, 2015). The empirical results and ablative studies show our method con- sistently achieves ofate-of-the-art performance on all benchmarks. In summary, our contributions are three-fold: (i) We exploit the CLIP knowledge to build a prior-enriched HOI representation. which is more robust for detecting fine-grained interaction types and under imbalanced data dis- tributions. (ii) We develop a self-taught relatedness classification loss to alleviate the problem of misaasoocattion between human-object pairs. (iii) Our approach achieves state-of-the -ftth----t- performance on the weakly-supervised HOI detection task on both benchmarks.
2 RELATED WORKS
HOI detection: Most works on supervised HOI detection can be categorized in two groups: two-stage and one-stage HOI detection. Two-stage methods first generate a set of homan-object proposals with an external object detector, then classify their interactions. They mainly focus on
exploring additional human pose information ( (Wan et al. 2019 Li et al. 2020a; Gupta et al. 2019) pairwise relatedness (Li et al. 2019aa] Zhou et al. 2020) or modeling relations between object and human (Gao et al., 2020: Zhang et al.) 202 Ulutan et al., 2020 Zhou & Chil 2019). to enhance the HOI representations. One-stage methods predict human & object locations and their interaction types simultancously in an end-to-end manner, which are currently dominated by transformer- based architectures (Carion et al.) 2020; Kim et al. 2022 Dong et al. 2022 Zhang et al. 202 b).
'Relatedness indicates whether human-object pair has relation, and interaction scores are multi-label scores on the interaction space.
2

---

Published as a conference paper at ICLR 2023
Object detector
feature map
HOI recognition network
HOI knowladge bank
Pg.
RM
W
HOl classification
Backbone
global branch
offlime
0.307 0443
fooding horse
Knowledge transfer network
Palrwise class classifiiattio sifficatiion network
IBAK pooling
outling cake
person (verb)
Textual Encoder
Vn Mp
0.2.8 ... 0336
(objent)
Rol-Alis
Roy Alian
driving
HOI prompt template
interaction classifiatioo
40
0
painwise
human feaftrre
objact feallure
global
self-laught relatedness dassifi-attoon
feature
feature
local branch
Figure 1: Model Overview: There are four modules in our network: backbone Network, HOI recognition
network, knowledge transfer network and a pairwise classification network.
Supervised methods show superior performance, but require labor-intensivy HOI annotations that are infeasible to obtain in many scenarios. Thus, in this work we focus on HOI detection under weak supervision.
Weakly-supervisee HOI detection: Weakly-supervisee HOI detection aims to learn instance- level HOIs with only image-level annotations. (Prest et al., (01T) learns set of binary action classifiers based on detected human-object pairs, where human proposal is obtained from a part- based human detector and object is derived from the relative position with respect to the human. PPR-FCN (Zhang et al.) 2017) employs parallel FCN to perform pair selection and classification. Explainable-HOI (Baldassarre et al., 2020) adopts graph nets to capture relations for better image- level HOI recognition, and uses backward explanation for instance-level HOI detection. MX- HOI (Kumaraswamy et al. 2021) proposes a mimeentttiiineeendeen learning strategy to utilize strong & weak labels simultaneously AlignFormer (Kilickaya & Smeulders 2021) proposes an align layer in transformer framework, which utilizes geometric & visual priors to generate pseudo alignments for training. Those methods focus on learning HOIs with advanced network structures or better pseudo alignments. However, they still suffer from noisy buman-object associations and ambiguous interaction types. To address those challenges, we exploit prior knowledge from CLIP to build discriminative HOI representations.
Knowledge exploitation of pretrained V&L models: Recently, CLIP (Radford et al. 202 la) model has demonstrated strong generalization to various downstream tasks Ghiasi al. 2021 Du et al. 2022 Gu et al. (02T). Some works also explore CLIP knowledge in supervised HOI detection, e.g., CATN (Dogg et al. (022)) initializes the object query with category-awarr aware semantic information (Liao et al. 2022) employs image feature distillation and classifier initialization with HOI prompts. However, they only exploit CLIP knowledge at a coarse level and require detailed annotations of human-object pairs. It is non-trivial to extend such strategies to the weak supervision paradigm due to highly noisy training signals. In our work, we build a deep connection between CLIP and HOI representation by incorporating the prior knowledge of HOIs both image and HOI instance levels.
from CLIP text encoder, and GEN-
3 METHOD
3.1 PROBLEM SETUP AND METHOD OVERVIEW
Problem setup Given input image the task of weakly-supervised HOI detection aims localize and recognize the human-object interactions, while only the corresponding image-level HOI categories are available for training. Formally, we aim to learn HOI detector M, which takes an image I as input and generates a set of tuples O = {(x,, Xo. Co, ar,o, R2,,)) }, i.c. O = M(I). Here each tuple indicates HOI instance, in which X E R represent human and object bounding boxes, c, E (1. C)) is the object category, On.o E A1) denotes the interaction class associated with X and x and Rgg IR is the HOI score. For the weakly-supervised setting,
3

---

Published as a conference paper at ICLR 2023
each training image is annotated with a set of HOI categories R {rr} at the image level only, where r" E {I....,N NN) is an index to a combination of ground-truth object category c" and interaction category a", and N denotes the number of all possible HOI combinations defined on the dataset.
Method Overview As we lack supervision for the HOI locations, we adopt typical hypothesize- and-recognize strategy (Zhang et al., 2017] Baldassarre al., [020; Kumaraswamy et al. (02T)) for HOI detection: first we generate a set of human and object proposals with an off-the-shel object detector (Ren et al.
2015) and then predict the interaction class for all human-object combinations.
Unlike other methods, we do not re-use the feature maps of the object or human detector we only keep the bounding boxes. Instead, we learn a new representation optimized for the HOI task. This is challenging under the weak setting as the model learning is noisy, but feasible by leveraging the rich semantic knowledge from a pretrained large-scale multimodal model, like CLIP. However, the naive knowledge integration strategies for supervised setting fail when directly applied in the weak setting, as evidenced by our experiments in Appendix B
Our framework adopts two philosophies to address the challenges in the veakly-supervin HOI task: the first is to integrate the prior knowledge into discriminative representation learning, and the second is to suppress noise in learning. For the first philosophy, we utilize the prior knowledge from CLIP to guide the representation learning in both global image-level and fine-grained human-object pairs, which is instantiated by a bi-level knowledge integration strategy. For the second philosophy, we adopt an effective telg-t learning mechanism to suppress the irrelevant pairs.
We instantiate the bi-level knowledge integration strategy with two-branch deep network. Our detection pipeline starts with a set of human proposals with detection scores {xx,, s,)}, and object proposals with their categories and detection scores {(x.> Cos s.)). Then, the global branch performs image-level HOI recognition by utilizing allows us to exploit both visual and text encoders from CLIP to generate better HOI representations. In parallel, for each human-object pair (X,, X.). the local branch explicitly augments the pairwise HOI features with the HOI knowledge bank to then identify their relatedness and interaction classes.
HOI knowledge bank as classifier. This
To train our model, we use multi-task loss, which incorporates a HOI recognition loss defined on image-wise HOIs for the visual encoder and knowledge bank finetuning, and a self-taught relatedness classification for suppressing the background human-object associations, top of the standard MIL-based loss. We first present model details in Sec 3.2 followed by the training strategy in Sec 3.3
3.2 MODEL DESIGN
Now we introduce our bi-level knowledge integration strategy, where the aim is to exploit CLIP textual embeddings of HOI labels as HOI knowledge bank for the HOI representation learning, and to transfer such knowledge both at image level as well as at the level of human-object pairs for interaction predictions. Specifically, as shown in Fig. [ our network consists of global branch and a local branch. The global branch includes a backbone network (Sec 3.2.Tp that extracts image features, and HOI recognition network (Sec (32.2] (3.23. that uses HOI knowledge bank based on CLIP to predict image-level HOI scores. For each human-object proposal generated by an off-the-shelf object detector, the local branch employs a knowledge transfer network (Scc(.22.3) to compute its feature representation with enhancement from the HOI knowledge bank, and a pairwise classification network (Sec to compute their relatedness and interaction scores. Finally, we generate the final HOI detection scores by combining global HOI scores with local predictions (Sec. 3.2.5)
HOI Knowledge Bank Generation CLIP builds a powerful vision- vision-language model by pretraining on large-scale image-text pairs. It consists of visual encoder Fy and textual encoder Fr, mapping both visual and textual inputs to a shared latent space. Here, we exploit CLIP to generate a HOI knowledge bank. We take a similar prompt strategy as in CLIP, adopting common template 'a person (verb) ) a/an (objecr)' to convert HOI labels into text prompts (e.g., converting 'drive car' to person driving car'). Then we input the sentences into the CLIP textual encoder FT to initialize the HOI knowledge bank Wr E IR N-D with D denoting the feature dimension. One can think of Wr as a set of 'prototypes' in feature space, one for each HOI in the dataset.
3.2.1 GLOBAL BRANCH: BACKBONE NETWORK
To incorporate CLIP for feature extraction, initialize the backbone network (e.g.. ResNet- 101 (He jet al. 2016)) with CLIP's visual encoder Fy to generate a feature map I for the input image I. We further compute a global feature vector vi RDD with self-attention operation (Radford et al. 2021b).
4

---

Published as a conference paper at ICLR 2023
horse
HCI label: feeding
car
0
argmax
artention
bus
0
Z
0
0
hya0
feature
011
044
m
088
0
m
0
Up
5. 04 m 0.3
feed
FCs
FCs
04 Uz ue 04 U.
(a) knowledge transfer network
(b) pseudo relatedness label generation
The knowledge transfer network explicitly transfers the discriminative relation-leva lovell semantic knowledge derived from CLIP to the pairwise HOI representations. Pseudo relatedness label generation uses the pairwise interaction scores to generate the pscudo association labels for self-taught relatedness classification
Figure 2:
3.2.2 GLOBAL BRANCH: HOI RECOGNITION NETWORK
We perform image-wise HOI recognition task with the HOI knowledge bank Wr. We obtain global HOI scores E IRN by computing the inner product between the image feature Vg and the knowledge bank Wyr Sg Wr l/g. where x is matrix multiplication. This has the effect of adapting the visual encoder and knowledge bank parameters to the HOI recognition task, fully taking advantage of the knowledge from CLIP
3.2.3 LOCAL BRANCH: KNOWLEDGE TRANSFER NETWORK
Given the CLIP-initialized visual encoder, a standard HOI representation can be formed by concate- nating the human and object appearance features along with their spatial encoding. However, even after the finetuning as described above, such a representation still mainly focuses on object-level semantic cues rather than relation-level concepts. In this module, we explicitly exploit the HOI knowledge bank WT to learn local relation-specine HOI representation. To achieve this, we propose an attention-based architecture as shown F().
Specifically, for each human proposal X₁2 and object proposal X.. we use Rol-Align (He et al. 2017) to crop the feature maps from T followed by self-attention operation to compute their appearance features Uh, Vo IR Then we compute spatial feature Usp by encoding the relative positions of their bounding boxes (Xn, x.) The holistic HOI representation Up E RRD is an embedding of the human and object appearance features and their spatial feature, i.e., Up F: (wn; toj Usp)), where
[:] is the concatenation operation and Fg multi-layer perceptron (MLP).
To enhance relation-level concepts, we further compute its union region X., E 4 (see Fig. 2 and extract the corresponding appearance feature v. IR via Rol-align over the feature map P. The union region important as it encodes relational context cues, but potentially also contains large amount of background that is noisy for model learning. We thus devise attention module that is similar in design to the HOI recognition network, but uses the union feature v, as query to extract meta-embedding Umetta E RD from the HOI knowledge bank Wr. The final HOI representation ip E R is built by fusing the holistic representation vp and Umeta with a MLP FK-
a Softmar X vw); Umeta = a x Wr: 8p FK + emeta).
Here a E N is the normalized attention weight and T is the transpose operation. Umeta encodes discriminative representation from CLIP and facilitates feature sharing between HOI classes.
3.2.4 LOCAL BRANCH: PAIRWISE CLASSIFICATION NETWORK
Given the relation- aware HOI representation é our final module performs a coarse-level clas- sification on human-object association and fine-level classification for interaction recognition. Specifically, we use two MI.Ps Fp and FB to predict the interaction scores 8p E R and the relatedness score sb E R for each human-object pair:
Sp Fr(G,):
St. =b Fn((,)
(2)
For details c.f. the appendix
5

---

Published as a conference paper at ICLR 2023
To train the model under weak supervision (see Sec. 3.3 scores into image-level interaction scores. Assume we have M pairs of human-object proposals for given image, and denote the interaction scores for the m-th pair as sp We first concatenate all the interaction scores to compose a bag S !sp:.... sp E R M.A, then we maximize over all pairs to obtain the image-wise interaction scores: 8p mar S.
further aggregate the pairwise interaction
3.2.5 MODEL INFERENCE
During model inference, we do not use the local interaction scores 8p directly. Instead, we normalize S with a Softmaz operation defined on all pairs: S = Softmar(S), and then compute the normalized
m
pairwise interaction scores ep = ((p)) Ep. where 8p is a row from S and a is Sigmoid function. This has the effect of measuring the contribution of a given pair, in case of multiple pairs in an image share the same interaction.
The final interaction score th.o for human-object pair (Xny xx) combines multiple scores, including the global HOI scores sg. the normalized pairwise interaction scores Ep. and the relatedness score sp- The overall HOI score o is combination of the interaction score and the object detection scores.
hoo = (3) .G (sb); R% = (sh. 80)7 -
where sg is the HOI score corresponding to a-th interaction and co-th object category in S.. eg the score of a-th interaction in Cp. and y a papee--arrmeter to balance the scores (Zhang et al. 2021c
Li et al. (019b))
3.3
LEARNING WITH WEAK SUPERVISION
To train our deep network in a weakly supervised setting. we use a multi-task loss defined on three different levels. Specifically, our overall loss function L consists of three terms: i) an image- HOI recognition loss C, to adapt CLIP features to the task of human-object interaction detection; ii) pairwise interaction classification loss Cp to guide the knowledge transfer towards fine-grained relation-aware representations; and iii) self-taught relatedness classification loss L, to prune non-interacting human-object combinations. Formally, the overall loss is written
L= Lg + Lp 1p+b Lo
(4)
Image-wise HOI recognition loss Ly: Given the HOI scores 8y and ground-truth HOI categories R. Lg is a standard binary cross-entropy loss for multi-label classification: L.g = [_BCE( R).
Pairwise interaction classification loss Cp: We adopt a MIL strategy that first aggregates the pairwise interaction scores and supervises this with image-level interaction labels as A = {a" } Given the image-wiss interaction scores Sp. C, is standard binary cross-entropy loss for multi-label classification as: Lp L (B,,
Self-taught relatedness classification loss C.: As human-object associations are not annotated, devise novel pseudo relatedness label generation mechanism for training self- self-taught binary classifier to identify valid human-object associations. Specifically, we observe that the human-object pairs with confident interaction scores are often associated after a short period of initial training without self-taught classification loss. Motivated by this, we use the interaction scores sp from the model under training supervise the relatedness classification.
Concretely, we generate pseudo labels B = {bi,..., ..., bm } for all human-object pairs in an image, where bm {0, 1} indicates the relatedness for the m-th combination. To this end, as illustrated in Fig 2bb). we first propose binary mask Z E {0, 1] M-A for all interaction scores S with respect to the ground-trutt object categories C {c*) }. For each human-object pair where the object label c, is included in C, we consider it as potential interactive combination and thus assign the corresponding row in Z as 1, and other rows as O. For the latter, we also immediately set 6n = O. Then we generate pairwise scores la E R-M for each ground-truth interaction a" by selecting the corresponding row from S @ Z. The pscudo label for the pair with the highest score is assigned as 1, i.c.. m. arg max t and bm. 1. We only select one positive pair for each a". Finally, Ls is m
defined as a binary cross-entropy loss: L. = Σ LBCE($T bm). where s. is the relatedness score m
for the m-th pair.
We also explore top-K selection in Appendix F

---

Published as a conference paper at ICLR 2023
Table I: mAP comparison HICO-DET and V-COCO test denotes the results not available. * stands for the method we re-evaluate with the correct evaluation protocol (see Appendix [ foor details) and meaans our re-implementation. For V-COCO, all object detectors are pretrained on MSCOCO dataset by default, and details about the evaluation metrics APS &&2 c.f. Appendix [I]
IN-IK denotes ImageNet with 1000 classes.
Methods
Backbone
Detecoor
Full
HICO DET (S)
Rare
Now-Rae
AP
AP
appendtrd
ICAN Klanetaal all
2014
INN-IIIOOOOOOOO
FRONN (0000)
14.84
10:45
16.55
45.5 30
$2.40
IMPNST Wow
RNSO-FVN (NN-KRCCDC))
FRONN (0000)
17.46
15.56
18.0
$2.00
TIN
Ligrar 20130
RNSO-FPN (IS-IK&CCCT)
FRONN (CO0O)
17.23
13.51
18:32
47.80
54.20
DJ RX 1IEe Jh MINM
RN50 IN (K&000O)
FRONN (COOO)
21.34
18.53
21.18
53.30
50.30
IDN
RN50 (NN (K80000)
FRONN (IIC DET)
26.29
22.61
27.39
53.30
00.30
SCO
KNS0-FYN (X-118/0000000)
FRONN (HICODDTT
$6.33
24.72
54.20
50.90
Kin 11119
TransSormer (N-IIACCCCC))))))
DETR OHIO-DDET)
25.10
17.34
27.42
40
QPNC
RNIO1= Tnaof-Taa (IN-11&&00000
DETR (COOO)
20.90
33.62
90
60.70
CATN
Trann-ran (IN-IK@HIC3-DETAAOOO
(HICO-DET)
31.86
83.84
Transfomncs (IN-IK&COCO)
DETR DETRCHCCO-DET)
51.17
DiTi
RNSO-TrannSa (IN-IK&COCO)
DETR CHICO-DET)
27.45
68.50
SSKT
Ridl-Tranffumrr (IN- K&CCOO))
31.34
67.30
YLK1 1400
RNI01 Thiss-Trnsffmmtt HIHHICO DET:
OHICO DET)
31.18
bemsum mpp72520 & Sontysormmeed mming. leunuing imtay-isnsl
AlinnFammt Kilickaya Sucallkan 2021
(HICO-DET)
16.34
ExplamatooHOO [Boliesarre at 2020
ResNeXti01 (N-IK&COOCO)
FRONN (COCO)
11:20
BO1
Kumser
RN101 IK&CCCO)
FRONN (COCO)
12.06
17.50
Zanng
dozaset)
(COCO)
17.55
dacaset)
KNI01 (CLPP
(00CO)
24.52
49.97
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets: We benchmark our model on two public datasets: HICO-DET and V-COCO. HICO-DET consists of 47776 images (38118 for training and 9658 for test). It has N 600 HOI categories, which are composed of C = 80 common objects (the same as MSCOCO (Lin et al., 2014 ) and A = 117 unique interaction categories. V-COCO a subset MSCOCO, consisting of 2533 images for training, 2867 for validation and 4946 for test. It has 16199 human instances, each annotated with binary labels for 26 interaction categories.
Evaluation Metric: Following (Chao et al. 2015), we use mean average precision (mAP) to evaluate HOI detection performance. human-objee pair is considered as positive when both predicted human and object boxes have at least 0.5 loU with their ground-truth boxes, and the HOI class is classified correctly.
4.2 IMPL. IMPLEMENTATION DETAILS
We use off-the-shel Faster R-CNN (Ren et al. 2015) pretrained on MSCOCO generate most 100 object candidates for each image. For V-COCO, it is worth noting that we train the object detector by removing the images in MSCOCO that overlap with V-COCO to prevent information leakage. The backbone network is initialized with the visual encoder from CLIP-RNI01 model and the feature dimension
1024.
For model learning, we set the detection score weight = 2.8 as default by following previous works (Zhang et al. 2021c; Li al., [2019b), then optimize the entire network with Adam W and an initial learning rate of 1e-5 for backbone parameters and le-4 for others. We detach the parameters of the knowledge bank on the local branch for better model learning. We train up to 60K iterations with batch-size 24 in each on 4 NVIDIA 2080TI GPUs, and decay the learning rate by 10 times in 12K and 24K iteration.
4.3 QUANTITATIVE RESULTS
For HICO-DET (Tab our approach outperforms the previous state of the arts on the weakly supervised setting by a clear margin, achieving 22.89 mAP with ResNet-50 and 25.70 mAP with ResNet-101 as backbone, For fair comparison, we also re-implement PPR-FCN with CLIP visual encoder. The results show that we still outperform PPR-FCN by sizeable margin, which validates the superiority of our framework. Besides, we even perform comparably with HOTR and IDN under an inferior experimental setting where HOTR adopts more advanced transformer encoder-decoder architecture, and both methods are trained with strong supervision. Furthermore, the mAP gap between Rare (training annotations 10) and Non-rare HOI classes in results is much smaller than other methods, demonstrating the superior generalization capability of our HOI representation for solving the long-tailed distribution issue. In detail, we achieve 0.62 mAP gap with Res
7

---

Published as a conference paper at ICLR 2023
Table 2: Ablation study HICO-DET dataset. "RNSO-FPN(COCOO" denotes the backhone initialized with Faster R-CNN parameters pretrained on MSCOCO dataset while "CLIP RN50" stands for the backbone initialized with CLIP visual encoder. Besides, we construct the knowledge bank Wr with random initialization,
or computing HOI prompts by RoBERTa or CLIP text transformer.
Methods
Backhone
Parameter intialiaation
knowledge bank
HOI recognition
Koowledee
KTN
score fusion
SRC
Full
mAA (25)
Rare
Non Rare
baseline
CLI' RN50
19.52
16.58
20.30
Exp 1
CLIP RN50
CLIP Text
20.31
18.34
20.90
ksp
CLIP RNS0
JP 'lext
(freeze Wiy
20:09
18.23
20.64
Eap
CLIP RN50
CLIP Text
20.86
18:40
21.60
Exp 4
CLIP RN50
CLIP Text
22.40
20.70
22.90
Exp
Exp 6
CLIP RN50
CLIP Text
20.75
19.38
21.16
Exp 7
CLIP RN50
CLIP Text
/
21.53
20.05
21.97
CLIP
CLIP
22.41
23.03
Exp
RNS0-FPN (COCO)
19.44
16.20
20.41
Exp 9
RN5O-FIN (COCO)
randomn RoBERTa
19.61
15.57
20.82
Exp 10
RN50-FPN (COCO)
20.45
16.46
21.65
and 1.53 with ResNet-101 backbone, which is much smaller than AlignFormer (3.14) and PPR-FCN (2.64), and supervised methods SSRT (9.01) and GEN-VLKT (4.9).
For V-COCO dataset, we report the performance of AProle in both scenariol and scenario2 for a complete comparison, which are 42.97 / 48.06 AProte with ResNet-50 and 44.74 / 49.97 AProte with ResNet- 101 as backbone. As shown in T our model achieves significant improvement compared with AlignFormer, and even is comparable with supervised methods TIN and iCAN.
4.4 ABLATION STUDY
In this section, we mainly validate the effectiveness of each component with detailed ablation studies on HICO-DET dataset. We use ResNet-50 as the backbone network to reduce experimental costs.
The baseline adopts the visual encoder from CLIP-RN50 to generate the vanilla HOI representation Up, which is directly used to predict the interaction scores Sy. Only pairwise interaction classification loss Cp used for model learning.
Baseline:
HOI recognition: We augment the baseline with a HOI recognition network and observe the full mAP improves from 19.52 to 20.41, as reported in Exp 1 of Tab. 2 It suggests that the learnable knowledge bank Wr serves as a powerful classifier to perform image-level HOI recognition and update the visual encoder for better HOI representation. We visualize the learned parameters of knowledge bank in Appendix [ to demonstrate its effectiveness. Furthermore, as in Exp 2, the performance slightly decreases from 20.31 to 20.09 when we freeze the training of the knowledge bank, indicating that joint learning of visual features and the knowledge bank is more appropriate for HOI detection.
Knowledge Transfer Network (KTN): KTN explicitly transfers the CLIP meta-knowledge to pairwise HOI features. As result, contributes 0.55 Full mAP improvement (Exp 3 v.s. Exp /) and most of the performance gains come from Non-rare classes. This result shows KTN is capable of extracting discriminative features from the relational knowledge bank to our HOI representation. We also study the effectiveness of the attention mechanism of KTN in Appendix B
Score fusion: In Tab. 2 we largely improve the Full mAP from 20.86 (Exp 3) to 22.40 (Exp 4) by fusing the global HOI scores 8, to pairwise interaction score Sp. As the HOI recognition network seamlessly inherits the visual-linguistic features from CLIP and directly adopts image labels as supervision, the global interaction scores pretty accurate and largely enhance the pairwise scores, demonstrating its strong capabilities to cope with long-tailed and fine- -meiinaieed HOI recognition.
Self-taught Relatedness Classification (SRC): Self-taught classification aims identify the relatedness between human and objects. The improvements from Exp 4 to ours show the effectiveness of our self-taught strategy, which is capable of figuring out the irrelevant human-object pairs and suppressing their interaction scores during inference.
The ablation results of Exp 5-7 in Tab. 2 show the KTN and SRC are able to facilitate each other. In detail, the SRC obtains 0.49 Full mAP improvement when the KTN is introduced (ours Exp 4), which is only 0.36 without KTN (Exp 5 v.s. baseline). Similarly,
Combining KTN& & SRC:
8

---

Published as a conference paper at ICLR 2023
(a)
(b)
C
WasT motorrydde
horse: 0.397,
an motarcydle: 0.515, 0.033, 0.550
dining, table: 0.006, 0.993,
ours: 0.38, 0.355, 0.998
ride_harse 0.405, 0,966, 0.998
stt di..ot_ddiinggatt table: 0.232, 0.953, 0.994
baselime: 0.0189
Lifers Good
LG
paint, Fire hydrant: aure 0.208, 0.505, 0.955 baseline: 0.0027
repair_ inspect truck: 0,48, 0.138, 0.979
stand skateboard: 0.009, 0.001, 0.58
hold lite: 0.039, 0. 0.892, 0.238
hold joteektte: D.78,, 0.992,0.995
Figure 3: Visualization of HOI detection results on HICO-DET test set. Red scores denote the negative HOI predictions. We mainly demonstrate the model's capabilities on four aspects: (a) coping with imbalanced HOI distribution; (b) distinguishing subtle differences among interaction types; (c) suppressing background HOI
classes, and (d) pruning irrelevant human object associations. The numbers reported are normalized pairwise interaction score, global HOI score and relatedness score.
the KTN contributes 0.78 Full mAP improvement with SRC (Exp 7 v.s. Exp 6), which is only 0.55 without SRC (Exp v.s. Exp l).
Our visual encoder and knowledge bank are both initialized from CLIP. We also explore different parameter initialization strategy in Exp 8-10. Specifically, we initialize the visual encoder with ResNet50-FPN pretrained on COCO detection task for the baseline (Exp 8), and the knowledge bank with random parameters (Exp 9) or embeddings of HOI labels from RoBERTa model (Exp 10) for the final model. We observe severe drops with all these initialization methods compared with ours, demonstrating the effectiveness and generalization ability of CLIP model. is worth noting that the mAP of Rare classes decreases from 16.20 in Exp 8 to 15.57 in Exp 9, which suggests the randomly initialized knowledge bank even aggravates the imbalance issue in final model.
Parameter initialization:
4.5 QUALITATIVE RESULTS
We show some qualitative results of our method in Fig 3 For each HOI prediction, we report (i) normalized pairwise interaction score, (ii) global HOI score and (iii) relatedness score for ours, and only pairwise interaction score for baseline. Fig a), ours interaction scores are more confident than baseline in Rare HOI classes, demonstrating the generalization ability of our CLIP-guidee HOI representation. Besides, when incorporating relational knowledge bank into pairwise HOI representation, our method is capable of distinguishing the subtle differences among similar HOIs in 23 v.s. inspect_truck:f truck:0.48 the bottom figure). Moreover, Fig the global branch suppresses background HOIs by predicting low global scores for them (c.g.. the global HOI score is 0.033 for sit_on_ motoooryye while the ground-truth is sit_on_ on_bicycle). Finally, in Fig.3(d), self-taugh relatedness classification strategy shows strong capability at recognizing the ambiguous human-object associations (e.g., 0.079 v.s. 0.994 in the upper figure).
Fig (e.g.,
CONCLUSION
In this paper, we propose bi-level knowledge integration strategy that incorporates the prior knowledge from CLIP for weakly-supervised HOI detection. Specifically, we exploit CLIP textual embeddings of HOI labels relational knowledge bank, which is adopted to enhance the HOI representation with an image-wise HOI recognition network and a pairwise knowledge transfer network. We further propose the addition of a self-taught binary pairwise relatedness classification loss to overcome ambiguous human-object association. Finally, our approach achieves the new state of the art both HICO-DET and V-COCO benchmarks under the weakly supervised setting.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
We acknowledge funding from Flemish Government under the Onderzoeksprogrammm. Artificiele Intelligentie (AI) Vlaanderen programme, Shanghai Science and Technology Program 21010502700 and Shanghai Frontiers Science Center of Human-centered Artificial Intelligence.
ETHICS STATEMENT
Hereby, we consciously assure that our study is original work which has not been previously published elsewhere, and is not currently being considered for publication elsewhere. We do not have ethics risks as mentioned in the author guidelines.
REPRODUCIBILITY STATEMENT
We use publicly available benchmarks, HICO-DET and V-COCO, to validate our method, Code is available at http://stthbb.commmmbaa111111999
REFERENCES
Federico Baldassarre, Kevin Smith, Josephine Sullivan, and Hossein Azizpour. Explanation-baser weakly-supervised learning of visual relations with graph networks. In ECCV, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. HICO: A benchmark for recognizing human-object interactions in images. In ICCV, 2015.
Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In WACV, 2018.
Leizhen Dong, Zhimin Li, Kunlun Xu, Zhijun Zhang, Luxin Yan, Sheng Zhong, and Xu Zou. Category-aware transformer network for better human-object interaction detection. arXiv preprint arXiv:2204.04911, 2022.
Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with Lision-language model. arXiv preprint arXiv:2203. 229..1444. 2022.
Chen Gao, Yuliang Zou, and Jia-Bin Huang. ican: Instance-centric attention network for human-object interaction detection. BMVC, 2018.
Chen Gao, Jiarui Xu, Yuliang Zou, and Jia-Bin Huang. Drg: Dual relation graph for human-object interaction detection. In ECCV, 2020.
Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung- Yi Lin. Open-vocabulary image segmentation. arXiv 2021.
/2. preprint 12.
Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. ICLR, 2021.
Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint arXiv:1505.0444 044744 2015.
Tanmay Gupta, Alexander Schwing, and Derek Hoiem. No-frills human-object interaction detection: Factorization, layout encodings, and training techniques. In ICCV, 2019.
Kaiming Hc. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In /CCV2017, 2017.
10

---

Published as a conference paper at ICLR 2023
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 961-970, 2015.
ASM Iftekhar, Hao Chen, Kaustav Kundu, Xinyu Li, Joseph Tighe, and Davide Modolo. What to look at and where: Semantic and spatial refined transformer for detecting human-object interactions. arXiv preprint urXi::2204.0000 007466, 2022.
Maximilian llse, Jakub Tomczak, and Max Welling. Attention- based deep multiple instance learning. In ICML, pp. 2127-2136, 2018.
Mert Kilickaya and Amold Smeulders. Human-object interaction detection via weak supervision. arXiv preprini arXiv:21/2.00492 2021.
Bumsoo Kim, Junhyun Lee, Jaewoo Kang, Eun-Sol Kim, and Hyunwoo J. Kim. Hotr: End-to-end human-object interaction detection with transformers. In CVPR, 2021.
Bumsoo Kim, Jonghwan Mun, Kyoung-Wone On, Minchul Shin, Junhyun Lee, and Eun-Sol Kim. Mstr: Multi-scale transformer for end-to-end human-object interaction detection. arXiv preprint vrXiv:2203.14709, 2022.
Suresh Kirthi Kumaraswamy, Miaojing Shi, and Ewa Kijak. Detecting human-object interaction with mixed supervision. WACV, 2021.
Yong-Lu Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang. Yan-Feng Wang, and Cewu Lu. Transferable interactiveness prior for human-object interaction detection. In CVPR, 2019a.
Yong-Lu Li, Siyuan Zhou, Xijie Huang, Liang Xu, Ze Ma, Hao-Shu Fang, Yanfeng Wang, and Cewu Lu. Transferable interactiveness knowledge for human-object interaction detection. CVPR, 2019b.
Yong-Lu Li, Xinpeng Liu, Han Lu, Shiyi Wang, Junqi Liu, Jiefeng Li, and Cewu Lu. Detailed 2d-3d joint representation for human-object interaction. In CVPR, 2020a.
Yong-Lu Li, Xinpeng Liu, Xiaoqian Wu, Yizhuo Li, and Cewu Lu. Hoi analysis: Integrating and decomposing human-object interaction. In NeurIPS, 2020b.
Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify associ- ation and enhance interaction understanding for hoi detection. arXiv preprint arXiv:2203. 13954, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
Wen Liu, Weixin Luo, Dongze Lian, and Shenghua Gao Gao. Future frame prediction for anomaly detection a new baseline. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Hitoshi Nishimura, Satoshi Komorita, Yasutomo Kawanishi, and Hiroshi Murase. Sdof-tracker: Fast and accurate multiple human tracking by kkipped-detection and optical- floow. arXiv preprint arXiv:2106.1 14259, 2021.
Guansong Pang, Cheng Yan, Chunhua Shen, van den Hengel Anton, and Xiao Bai. Self-trained deep ordinal regression for end-to-end video anomaly detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2020.
Alessandro Prest, Cordelia Schmid, and Vittorio Ferrari. Weakly supervised learning of interactions between humans and objects. IEEE TPAMI, 2011.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021a.
11

---

Published as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021b.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. arXiv preprini urXiv: 2015.
Masato Tamura, Hiroki Ohashi, and Tomoaki Yoshinaga. Qpic: Query-based pairwise human-object interaction detection with image-wide contextual information. In CVPR, 2021.
Tina, Anmol Kumar Sharma, Siddharth Tomar, and Kapil Gupta. Various approaches of human activity recognition: A review. In International Conference on Computing Methodologies and Communication 2021.
Oytun Ulutan, A S M Iftekhar, and B. S. Manjunath. Vsgnet: Spatial attention network for detecting human object interactions using graph convolutions. In CVPR, 2020.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. JMLR, 2008. URL http:// r. org. pwwppappaapaaaa httppp/rrrs/.//vvvveeeaaaaaattte vaandermattee@8a html
Mrabti Wafae, Baibai Kaoutar, Bellach Benaissa, Oulad Haj Thami Rachid, and Tairi Hamid. Human motion tracking: A comparative study. Procedia Computer Science, 148:145-1533 2019.
Bo Wan, Desen Zhou, Yongfei Liu, Rongjie Li, and Xuming He, Pose-aware multi-level feature network for human object interaction detection. In ICCV, 2019.
Aixi Zhang, Yue Liao, Si Liu, Miao Lu, Yongliang Wang, Chen Gao, and Xiaobo Li, Mining the benefits of two-stage and one-stage hoi detection. NeulPS, 2021a.
Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Efficient two-stage detection of human- object interactions with a novel unary- mairr-ieirwise transformer. arXiv preprint arXi::2112.01839 2021b.
Frederic Z Zhang, Dylan Campbell, and Stephen Gould. Spatially conditioned graphs for detecting human-object interactions. In ICCV, 2021c.
Hanwang Zhang, Zawlin Kyaw, Jinyang Yu, and Shih-Fu Chang. Ppr-fen: Weakly supervised visual relation detection via parallel pairwise r-fen. In ICCV, 2017.
Desen Zhou, Zhichao Liu, Jian Wang, Leshan Wang, Tao Hu, Errui Ding, and Jingdong Wang. Human-object interaction detection via disentangled transformer. arXiv preprint arXiv:2204.09290, 2022.
Penghao Zhou and Mingmin Chi. Relation parsing neural network for human-object interaction detection. In ICCV, 2019.
Tianfei Zhou, Wenguan Wang, Siyuan Qi, Haibin Ling, and Jianbing Shen. Cascaded human-object interaction recognition. In CVPR, 2020.
12

---

Published as a conference paper at ICLR 2023
APPENDIX
In this appendix, we first describe the spatial feature generation, and then supplement more ex- perimental results of different CLIP knowledge integration strategies for weakly-superviser HOI detection. For Explanation-HO1 (Baldassarre et al., 2020), we further clarify the difference between their mAP evaluation protocol and the standard one. Finally, we demonstrate the limitations, potential negative societal impacts as well the result error bars of our method.
A THE ADVANTAGE OF OUR HOI REPRESENTATION
To verify the improvement obtained with our CLIP-based HOI representation, we visualize the HOI representation ip in feature space with t-SNE(van der Maaten Hinton, 2008). For clarity, we randomly sample 80 HOI categories, and collect 30 samples for each category. For comparison, we also demonstrate the object-baset based HOI representation derived from 'Exp in Tab.2 (i.e., the model without CLIP knowledge and using random knowledge bank). As shown in Fig/4] we observe that CLIP-based HOI representations for different HOI categories are diverse and well separated in feature space, which is better for HOI detection. In contrast, the object-based representations are not well separated in feature space (see the red box region in F) Besides, the experimental results in the ablation study (ours v.s. 'Exp 9¹) also validate the advantage of CLIP-based HOI representation, improving full mAP from 19.61 to 22.89.
B ABLATION ON CLIP KNOWLEDGE INTEGRATION
To further demonstrate the superiority of our CLIP knowledge integration strategy, we study several proven techniques for CLIP knowledge transfer in Tab. 3 In Abl 1, for each human-object pair, we directly infer the HOI scores with CLIP by computing the cross-modal similarities between their visual union region and the HOI prompts. Without introducing any HOI priors, the promising results indicate the powerful generalization ability of CLIP and motivate the design of incorporating CLIP knowledge for weakly-superviie results from Exp 8 in Tab. 2 of the main paper. It simplified baseline model but initializes the visual encoder with ResNet50-FPM pretrained on COCO detection task. Then we introduce three different CLIP knowledge transfer strategies (Abl 3-4 and ours) based on Abl
HOI detection. In Abl 2, we duplicate the experiment setting and
In Ab/ 3, we directly enhance baseline scores in Ab/ 2 with the CL.IP similarity scores in Abl / on the inference stage. Without bells and whistles, we obtain 1.12 gain in Full mAP.
Furthermore, in Abl 4, we adopt a similar knowledge transfer strategy as GEN-VLKT (Liao et al. 2022), where we initialize the HOI classifier Fp with HOI prompt and regularize the global HOI representation with CLIP image feature vig. In detail, we first compute the global HOI representation Umean with mean pooling on all pairwise HOI representations, i.e., Uneen = Mean MeanPooll (1.) Im=1)
m
Here w is the holistic HOI representation (c.f. Sec. 3.2.3 in the main paper) for m-th human-object pair. Then we develop an additional L2 loss Creg transfer the knowledge from CLIP HOI representations: Lreg = L2( L2memn vgy) The performance even decreases slightly from 19.44 to 19.39, which might be caused by the incompatibility of parameters between backbone network (ResNet50-FPN pretrained on COCO) and Fp (HOI prompt embeddings from CLIP). When directly applying the knowledge transfer strategy of to map the unmatched HOI representation and classification weights to joint space as the supervisory signals are noisy.
to a weakly supervised setting, it difficult
Finally, our approach achieves the best performance compared with other strategies, demonstrating the effectiveness of our bi-level knowledge integration strategy.
C SPATIAL FEATURE GENERATION
Following Zhang et al. 2021c). generate the spatial feature Usp R for each pair of human- object proposals (X,, X,). Specifically, we first compute the bounding boxes information for X;, and x, separately, including their center coordinates, widths, heights, aspect ratios and areas, all
13

---

Published as a conference paper at ICLR 2023
Sall-boat
80
40
book
dapoop
molorcycle
20
swing
baseball
call di.diniigg table
-20
5it at-dininng table
-40
-40
bird
cosch
-60
-40
-20
0
20
40
60
-50
-40
-20
20
40
60
90
CLiP-based representations
object-based representatios (Tab22-Exp5)
Figure 4: The t-SNE visualization of CLIP-based HOI representation and object- HOI represen- tation.
Table 3: Ablation of different CLIPP knowledge integration strategies on HICO-DET datasct.
Methods
Experimental setting
MAP(S)
Abl ]
CLIP inference score
11.84
13.72
11.27
2
RNS0-FPN (COCO) + 'Fp random
16.20
Abl 3
RNS0 FPN (COCO) + Jp random init. CLIP inference scure
20.56
18.19
21.27
RN50-EPN (CUCO) prompt init.
visual regularization
19.39
ours
RN50 + 1101 recognition KTN sell taught relatedness cls.
22.89
22,41
23.03
normalized by the corresponding dimension of the image. We also encode their relative spatial relations by estimating the intersection union (IoU), ratio of the of x₁, and X,, directional encoding and the distance between center coordinates of x₁, and x We concatenate all the above- mentioned preliminary spatial cues and obtain spatial encoding p E R...S. To encode the second and higher order combinations of different terms, the spatial encoding is concatenated with its logarithm and then embedded to usp Usp FFp (p: log(p + +e]]). Where 0 is small constant guarantee the numerical stability, and F sp is multi-layer fully connected network.
D VISUALIZATION OF HOI KNOWLEDGE BANK WT
To further understand Wr, we visualize the knowledge bank features initialized by CLIP (Fig.5(a)) and learned from scratch (Fig.5(b)) in feature space by t-SNE. It is worth noting that the knowledge bank learned from scratch is derived from 'Exp 9' in Tab.2. As shown Fig(5] we observe that the knowledge features of HOI classes initialized with CLIP are more discriminative than random initialized, and show better clustering result (e.g. the HOI classes in red box regions).
E DIFFERENT DESIGNS OF KTN
To further validate the effectiveness of our attention mechanism in KTN, we compare our design with some variants in Tab. 4 First of all, we directly encode the relation-level features within the union region to enhance the pairwise representation rather than the external knowledge bank. As result, the mAP even decreases a little bit from 20.75 (Exp 6) to 20.69 (Exp 1I). The potential reason is that the union region contains more ambiguous visual relations and background clutters, which are difficult to learn in a weak setting. Besides, we also explore different normalization strategies in demonstrate that Softmar operation (ours) performs better than uniform
KTN. The results in Tab. 4 attention (Exp 12) or Sigmoid operation (Exp 13), indicating our attention mechanism is non-tre trivial and more effective aggregating the relational cues from HOI knowledge bank.
14

---

Published as a conference paper at ICLR 2023
(a) Ours knowledge bank
Seree on Exomkneennig dee.e
25
so interaator depare
14
Furandioo
-15
-25
-35
-0
-20
-10
10
15
Pepecterping
5
bourd buin
bastinn
-5
-15
Lan
Iepastoou
-30 -20 -10
20
(b) Knowledge bank learned from scratch
Figure 5: The t-SNE visualization of knowledge bank Wy. (a) is the knowledge bank distribution in feature space based on our CLiP-based HOI representation while (b) the knowledge bank learned from scratch (the model in Tab.2-Exp 9) based on object-based HOI representation.
Table 4: Different network design of Knowledge Transfer Network (KTN).
Methods
Parauecter mntiaaizaiiin
CLIP Know Knowledge
SRC
mAP (5)
knowledge
recognition
Rare
Exp 77
CLI RN50
CLIP Text
(wnion)
20.69
19.55
21.04
Exp 12
RN50
Text
(uniform) (sigmoid)
19.82
21.53
Exp 13
CLIP RN50
CLIP Text
21.28
19.27
21.88
60rs
CLIP RN50
CLIP Text
21.53
20.05
21.97
F TOP-K POSITIVE PAIR SELECTION FOR SRC
In this section we show the results of selecting top-2 and top-5 pairs as positive in Tab We notice that there is small performance drop. which is likely to be caused by mislabeling more negative pairs as positive, resulting in model learning with more noise.
G THE PROMPT GENERATION FOR V-COCO
For the V-COCO dataset, each action has two different semantic roles ('instrument' and 'object') for different objects, like "cut cake' and 'cut with knife', We use two different prompt templates to convert a HOI label to a language sentence. For the former one, we take template "a person verb a/an object", and use person verb with object" for the latter.
H EVALUATION METRIC FOR V-COCO
V-COCO dataset has two scenarios for role AP evaluation. In Tab. J APSI&2 refer to 'Average Precision in scenario 1&2'. V-COCO dataset has two different annotations for HOIs: the first is
15

---

Published as a conference paper at ICLR 2023
Table 5: Ablation of top-K positive pair selection for SRC on HICO-DET dataset.
mAP (%)
Methods
Full
Rare
Non-Rare
Top-3
22.45
21.61
22.70
Top-2
22.49
21.83
22.69
(Top-1)
22.89
22.41
23.03
Iel CL SIETLER PELERNE pIs DI1 1OL
210
with duss Parts DE1 TOAA www.ooo coordrr Tar
MaaLos (6c181) pretute Badrsl dap Deolat1
ten termuas win
212
torch bensen pek mos bu setsatt)
conter peuutt, reclectt
21a
215
234
217
astons retl tart aooi, 15MLLa0 Ailoeeee
Men, F1. statior... (lumericase))
22
For enul mecn targ 200
Lecject 8.31
Break
Nit. Pat sboes Pestast porm onsenction or Aol sorres adi
Traustaa's
(a) Evaluation protocol in Explanation-HOl
(b) The correct evaluation protocol
Figure 6: The screenshot of the evaluation code in Explanation-HOI. (a) is the original code while (b) the correct one based on the standard evaluation code. We use red rectangle boxes to highlight the most important differences
full label (human location, interaction type, object location, object type), and the second misses target object (also denoted as 'role' in the original paper (Gupta & Malik 2015) annotations, and the label only includes (human location, interaction type). For the second case, there are two different evaluation protocols (scenarios) when taking a prediction interaction is correct & the overlap between the human boxes > 0.5 & the corresponding role is empty, which is more restricted: in scenario only requires the interaction correct & the overlap between the person boxes is >> 0.5.
correct In scenario it requires the
I EVALUATION OF EXPLAVATION-HOL
The Explanation-HON (Baldassarre et al. 2020) has misunderstanding of mAP evaluation protocol. As shown in Fig ([a) L200--205, the Explanation- HOI only takes some specific predicted HOIs into the evaluation process, which has the same HOI labels as groundtruth HOIs. Thus, they ignore lots of false-positive HOI predictions when calculating mAP, leading to an untrustable high mAP score evaluate all predicted HOIs, which is
(reported in their original paper). In Fig/6(b) I. 1.204-1.208, the same as the standard evaluation protocol proposed in HICO-DET (Chao et al.) 20155). The correct results have already been reported in Tab. the main paper.
LIMITATIONS
As described Sec. 3.1, we adopt an external object detector to generate human-object proposals and then recognize their interactions. Consequently, our method is faced with two limitations brought by erroneous object detection results. Firstly, the positive human-object pairs are not recalled if the human object proposals not detected. Secondly, the proposals kept fixed during learning, which leads to the problem of inaccurate localization and object types.
16

---

Published as a conference paper at ICLR 2023
K RISK OF USING CLIP
For all the methods that adopt CLIP in their model design, there is potential risk of data leakage as CLIP has seen quite lot of data during pretraining. For HOI detection task, cannot get access CLIP dataset and do not know the exact overlap between CLIP and HOI benchmarks (i.e., HICO-DET and V-COCO), we carefully read Sec. (Data Overlap Analysis) of the CLIP paper (Radford et al. 202Ib), including an analysis of the overlap between its dataset with 35 popular datasets (HICO-DET and V-COCO are not included). shows the overlap is small (median 2.2% and average is 3.2%) and the influence is limited ("overall accuracy is rarely shifted by more than 0. % with only 7 datasets above this threshold"). Besides, the training text accompanying an image in the CLIIP dataset is often not related to the HOI annotations. Thus, we think the risk is limited.
L LICENSE
The licenses of the assets used in our work are listed below, including open-sourced CLIP model, HICO-DET dataset, and V-COCO dataset. As for HICO-DET, we cannot find its license the paper and the official project page. Thus provide the official project page instead here for clarity.
1. CLIP: httpss//ggibab..cooooppaaiCCCL MIT License
2. VCOCO: https//ggthabb.oom/ggggua/voooo MIT License
3. HICO-DET:
ywchao/hicol No license
17


Reviews:
[{"review_id": "otcC-Au4uv", "reviewer": "Reviewer_qCHU", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a new weakly supervised method in the human-object interaction task. The method exploits the prior knowledge from CLIP.", "summary_of_the_review": "The paper presentation is bad which creates troubles for me to fully understand the method section.\nThis subsequently creates problems for me to appreciate the novelty of the method.\nI am also having issues with the baseline comparisons. I do not think it is fair to some extent (see elaborations above).\n\nI would vote for REJECT. However, it is possible that I might revise my ratings based on the authors' feedback after the rebuttal period.", "strength_and_weaknesses": "[Strength] \nThe idea of incorporating external knowledge from CLIP is interesting. \n\n[Weakness]\n1. The authors promise to release source codes. This would eliminate the issue of reproducibility. However, the paper presentation is bad, which makes readers hard to fully understand and hence, potentially hard to reproduce the model based on the main text.\n\n2. The paper presentation is bad (see more points below).\n\n3. There exists an unfair comparison with baselines in the experiments (see elaborations below).\n\nI elaborated on the specific points below:\n[clarity] \nThe paper presentation is very bad. I have trouble understanding some parts of the model designs and the problem setting. The authors are strongly encouraged to explicitly list out what are the supervised signals in the problem setting in both fully supervised and weakly supervised cases. Please clearly define every math notation before using them. I list out several key confusions:\n\n1.1. What is the detection score s_h defined in sec 3.1? Ground truth bounding box coordinates? How many scores are there?\n\n1.2. Define relatedness and interaction scores in the introduction. \n\n1.3. Add vg defined in sec3.2.1 to fig1. Also including several other key variables in Fig1 would help readers understand the designs better.\n\n1.4. I have trouble understanding this relatedness score. Up to 3.2.5, there is no definition of sb. What is the relatedness score exactly? Is this binary score given as ground truth? If so, the authors should spell it out at the beginning of the paper.\n\n1.5. In sec3.3, it talks about self-taught relatedness classification loss. What is exactly given in this case? If binary cross-entropy loss is used and bm is given, this relatedness acts as the ground truth and it should be explicitly introduced in the problem setting in Sec 3.1.\n\n1.6. what is given in the weakly supervised setting? What are the image-level annotations only? Go back to the tuples O defined in Sec3.1, what is exactly missing or not given in a weakly supervised setting?\n\n\n1.7. In table1, what are APS1 and S2 in the rightmost column?\n\n[other questions]\n2. I am familiar with scene graph generation. HOI seems to be a subset of that problem. Would this weakly supervised method generalize to scene graph generation?\n\n3. I found the comparisons of methods are unfair and the results are puzzling. In table 1, baselines are using different backbones and object detectors. If so, it is very hard to say whether the performance improvement is simply because of the change of backbones and detectors, or because the proposed weakly supervised methods are efficient.\n\n4. Utilizing the external knowledge from CLIP is an interesting idea! However, would this put other methods at disadvantage as CLIP has been presented with more data than existing methods?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "m-UNBWXKjJe", "reviewer": "Reviewer_q7LZ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a bi-level knowledge integration strategy that incorporates the prior knowledge from CLIP for weakly-supervised HOI detection.\n\nThis paper also exploits CLIP textual embeddings of HOI labels as a relational knowledge bank, which is adopted to enhance the HOI representation with an image-wise HOI recognition network and a pairwise knowledge transfer network.\n\nThis paper further proposes the addition of a self-taught binary pairwise relatedness classification loss to overcome ambiguous human-object association.\n\n", "summary_of_the_review": " It lacks any innovative contribution.", "strength_and_weaknesses": "The proposed approach achieves the new state of the art on both HICO-DET and V-COCO benchmarks under the weakly supervised setting.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "biZ1teIVPyQ", "reviewer": "Reviewer_frLC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work addressed the problem of HOI detection tasks under a weakly-supervised learning paradigm. The key inspiration is to leverage the rich vision-language model (i.e., CLIP) to improve the model learning on highly noisy and long-tailed HOI datasets. The key novelties are the novel local relation-specific HOI representation and the relatedness classification terms. The experimental results show that the proposed approach improves the corresponding baseline. ", "summary_of_the_review": "Overall, this paper is well-written and easy to follow. Generally, the work is highly beneficial from the strong visual-language model, where the object semantics are better learned in the pre-trained model. The key contribution of this work is to better exploit CLIP embedding for learning relational information. The designed model is intuitive and provides insights on how to leverage such information for a specific task. ", "strength_and_weaknesses": "Strength:\n+ This work acknowledges the rich information encoded in the pre-trained CLIP model and proposes several modules to better leverage such information for the HOI detection task. This include (1) HOI knowledge bank, and (2) knowledge transfer network. \n+ Design a local relation-specific HOI representation that transfer relation-level semantic knowledge from CLIP to pair-wise representations. \n+ Design a model-guided relatedness classification loss term (i.e., some form of trustworthiness score) to address the problem in weakly supervised learning. \n+ The paper conducted several ablation studies to empirically show the efficacy of the proposed modules. Improvement on rare classes is more significant, demonstrate the benefit of having strong pre-trained model.\n\nWeakness:\n- The technical novelty of this work is relatively weak. The relatedness classification loss term is very straightforward and used in earlier work (e.g., using a threshold to remove low-quality prediction). In addition, this work largely benefits from the strong CLIP model (e.g., HOI knowledge bank, visual encoder, and text encoder). Effectively, the long-tailed issue is partially addressed as the encoder is trained with external dataset. The strength is that this work shows how to better leverage the CLIP model in the context of HOI detection. But I would argue that the technical novelty is welcome but limited.  \n\nOther concerns:\n- Introduction highlights that JOI annotations are prone to image-level labelling errors. Can the author elaborate on how can the model deal with labelling errors if it is presented in training data? \n- In Fig 1, please clearly annotate which are the HOI recognition network (is it both local and global branch?)\n- In Section 3.2, \"The global branch includes a backbone network (Sec.3.2.1) that generates human-\nobject proposals and extracts image features\". The human-object proposals seems to be generated with a pre-trained object detector and provided as an input to global brunch, please clarify this. \n- Please comment about the baseline model in ablation study. It seems that the performance is actually higher than the PPR-FCN (which use the same backbone) in Table 1. Is there some clear factors that give the baseline a higher performance?\n- Please change \"strongly supervised learning\" to \"supervised learning\". The term is well established in the community in the past decades. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "S2_FCCIwpuo", "reviewer": "Reviewer_RZBb", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "For weakly-supervised HOI identification, this paper provides a bi-level knowledge integration technique that integrates the prior information from CLIP. To augment the HOI representation using an image-wise HOI recognition network and a pairwise knowledge transfer network,  the authors specifically use CLIP textual embeddings of HOI labels as a relational knowledge bank. To get over unclear human-object connection, the authors also suggest including a self-taught binary pairwise relatedness classification loss. Finally, in a weakly supervised context, the proposed method surpasses the previous state of the art on the HICO-DET and V-COCO benchmarks.", "summary_of_the_review": "This paper looks good to me. The main concern is that CLIP text prior seems not to be strongly related to weakly supervised learning considering pseudo labels contribute only a little improvement.\n", "strength_and_weaknesses": "Strength\n- The method achieves the sota performance among weak-supervised hoi detection benchmark.\n- Utilizing the prior knowledge from encoded prompt feature bank to enhance the human-object feature is interesting\n\nWeaknesses\n- Literature review should be enhanced, e.g., [1].\n[1] Weakly Supervised Learning of Interactions between Humans and Objects\n- From my humble view,  the score fusion is much better since union region brings lots of noise. Therefore, using the union region feature as  query to get attention score still be noisy.\n- It seems that over 50 percent images of HICO—DET only contains one human-object pair (pls correct me if wrong). Therefore, HICO-DET is probably not the best choice to validate idea, especially for ablation study. \n- From my prospective, the main contribution of the paper is exploring how to introduce CLIP text prior but it is not strong related to weakly supervised learning. How about the result if taking into account fully supervision? \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FHX-vXV9CU9", "reviewer": "Reviewer_S5GP", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper targets on HOI detection in a weakly supervised manner (i.e., only image description is provided, no instance-level annotation). To improve the generalization towards fine-grained human-object interactions and mitigate the biased learning caused by imbalanced data distribution, this paper proposes a training strategy to distill the knowledge from CLIP model. Specifically, they incorporate CLIP in HOI at both the image and the instance levels. At the image level, they generate a global HOI knowledge bank and predict image-level HOI descriptions. At the instance/object level, the attention mechanism is employed to enrich the representation of HOI features and then improve the performance. A comprehensive evaluation is conducted and a detailed analysis is provided. ", "summary_of_the_review": "The overall presentation of this paper is good. However, a few more discussions should be provided. Please see details in weakness.", "strength_and_weaknesses": "Strength:\n1.\tThe motivation is clear. Applying CLIP in weakly supervised learning and facilitating the learning of an HOI detector is essential. \n2.\tThe approach is interesting. Though combining the detector with CLIP has been studied, but combining them for weakly-supervised learning has been less explored.\n3.\tThe experimental study is promising and comprehensive. The visualizations are also provided.\n4.\tThe ablation study is in detail.\n\nWeakness:\n1.\tRegarding the whole framework, which part is vital for using CLIP to guide weakly supervised learning? I think the discussion is necessary (but I didn’t find clear answer in the discussion) and help this paper to be distinguished from the other related work.\n2.\tThe knowledge bank is based on classes appearing in the full dataset and is defined by the text. Can you explain how the size of the knowledge bank affects performance? After all, when there exists a brunch of interaction classes, I am not sure about the training efficiency and workload.\n3.\tSRC only does not help much with the detection, according to Table 2. It is not very effective and is kind of counterintuitive. I would recommend providing a more detailed explanation or removing the SRC part.\n4. When a CLIP model is used, it is always necessary to explain the potential issue of fair comparison. After all, CLIP has seen quite a lot of training data during pretraining, and there is a risk of potential data leakage. As such, explanation is necessary.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster