Towards Flexible Visual Relationship Segmentation
Authors: Fangrui Zhu
Jianwei Yang
Huaizu Jiang
Venue: NeurIPS 2024 poster (2024)
Keywords: Visual relationship segmentation
relationship understanding
human-object interaction
scene graph generation.
Primary area: machine_vision

Abstract: Visual relationship understanding has been studied separately in human-object interaction(HOI) detection, scene graph generation(SGG),  and referring relationships(RR) tasks. 
Given the complexity and interconnectedness of these tasks, it is crucial to have a flexible framework that can effectively address these tasks in a cohesive manner.
In this work, we propose FleVRS, a single model that seamlessly integrates the above three aspects in standard and promptable visual relationship segmentation, and further possesses the capability for open-vocabulary segmentation to adapt to novel scenarios. 
FleVRS leverages the synergy between text and image modalities, 
to ground various types of relationships from images and use textual features from vision-language models to visual conceptual understanding.
Empirical validation across various datasets demonstrates that our framework outperforms existing models in standard, promptable, and open-vocabulary tasks, e.g., +1.9 $mAP$ on HICO-DET, +11.4 $Acc$ on VRD,  +4.7 $mAP$ on unseen HICO-DET.
Our FleVRS represents a significant step towards a more intuitive, comprehensive, and scalable understanding of visual relationships.

Towards Flexible Visual Relationship Segmentation
Fangrui Zhu'
Huaizu Jiang' Northeastern University "Microsoft Research https://neut github io/FleV
Jianwei Yang²
Standard HOI Seymentation
Promptable HGI Segmentatton
Surson stradello, hurseo
<person walk, horse
sperson, ride, horse>
Promet: stradiie,
hold, horses
Prompt: niee, horse> <peison
<person Norsas
horse>
Promptable Panoptic SGG
Standard Panoptic SGG
<punch
paneerent
<peroon
bencho
look o5, porSOn
bench
<berch,
cherch on, pasement
Spenson
pavements
Figure 1: FleVRS is single model trained to support standard, promptable and vocabuacyy fine-grained visual relationship segmentation (<subject mask, relationship categories, object mask>). It can take images only or images with structured prompts inputs, and segment all existing relationships or the ones subject to the text prompts.
Abstract
Visual relationship understanding has been studied separately in human-object interaction (HOI) detection, scene graph generation (SGG), and referring relation- ships (RR) tasks. Given the complexity and interconnectedness of these tasks, it is crucial to have a flexible framework that can effectively address these tasks in a cohesive manner. In this work, propose FleVRS, single model that seamlessly integrates the above three aspects in standard and promptable visual relationship segmentation, and further possesses the capability for open-vocabulary segmenta- tion to adapt to novel scenarios. FleVRS leverages the synergy between text and image modalities, textual features from vision-language models to visual conceptual understanding. Empirical validation across various datasets demonstrates that our framework performs existing models in standard, promptable, and vven-vocabulary tasks, e.g., +1.9 mAP on HICO-DET, +11.4 Acc VRD, +4.7 mAP on unseen HICO-DET. Our FleVRS represents a significant step towards a more intuitive, comprehensive, and scalable understanding of visual relationships.
ground various types of relationships from images and
Introduction
image not merely a collection of objects. Understanding the visual relationships between
different entities at pixel-level through segmentation is fundamental task in computer vision, which
has broad applications autonomous driving [28 (88, behavior analysis [65] 67], navigation [10 [5]
22 27), etc. Furthermore, segmenting relational objects extends beyond mere detection, playing
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Standard
Method
Promptahle
Open-vocabulary
One/Two-stage Model
HoI
SGG
RLIPv2 92 UniVRD
x
Two
101
x
X
Two
SSAS
38
X
X
V
X
One
GEN-VLKT
47
X
X
One
FLeVRS (Ours)
V
One
Table 1: Comparisons with previous representative methods in three aspects of model capabili-
ties. To the best of our knowledge, our FLeVRS is the first one-stage model capable of performing standard, promptable, and open- open-vocabulary visual relationship segmentation all at once.
crucial role in improving visual understanding and providing a more comprehensive abstraction on the visual contents and interactions among them.
Ideally, visual relationship segmentation (VRS) model should demonstrate flexibility across three key dimensions. 1) Capability of segmenting various types of relationships, including both human-centric and generic ones. These relationships are defined as triplets in the form of <subject, predicate, object>. Human-object interaction (HOI) detection [4 [8), which we adapt into HOI segmentation in our work, exemplifies this capability, such as <person, ride, horse> in Fig. Panoptic scene graph generation (SGG) [55] 81 [5]. captures generic spatial or semantic relationships among pairs of objects in an image, e.g., bench on pavement in Fig. [T] A unified model that can handle these tasks concurrently is essential, as it eliminates the need for separate designs and modifications for each specific task. 2) Grounding of relational subject-object pairs with different prompts. Given various textual prompts, the model should output the desired entities and relationships, facilitating a more natural and intuitive user interface. For instance, it should be able to detect just the person in image all possible interactions between a person and a horse, as illustrated in Fig. [ 3) Open-vocabulary recognition of visual relationships. In realistic open-world applications, the model should generalize to new scenarios without requiring annotations for new concepts not seen during training. This capability includes detecting novel objects, relationships, and their combinations.
Existing models in visual relationship segmentation (VRS) have targeted aspects of the desired capabil- ities but fall short of providing comprehensive solution, as detailed in Tab. [ Models have typically
focused on tasks like human-object interaction (HOI) detection [23 35|
47
72 95| 97 [08] and
panoptic SGG (50 71
81
85
X
104.
Although models such [92 101 have attempted to unify
VRS under a single framework, they need additional pretraining on HOI datasets (Tab. [ and lack features such promptable segmentation, which allows for dynamic entity and relationship genera- tion based on textual prompts, as well as capabilities for open-vocabulary promptable segmentation. Efforts to detect instances referred to by textual prompts have been made [21 38 75 [07], but these models fail to capture all desired entities or relationships comprehensively and struggle with classi- fying multi-label interactions between the pairs, limiting their effectiveness in complex scenarios. Although recent models such as (2 5 76 78 88) exhibit enhanced capabilities grounding instances specified by free-form text and show strong generalization over novel concepts, they still do not generate the required pairs in the format of segmentation masks. Furthermore, these models require significant computational resources and additional vision models for precise localizations. For open- vecabuacay VRS, existing works [7 91 92 leverage textual embeddings to transfer knowledge. However, models 99] (2 fall short in grounding diverse prompts, while 47 is exclusively designed for HOI detection, not generic VRS.
grounding models like (29] 52 83 and multimodal large language
To address the limitations in existing models, we introduce FLeVRS, a flexible one-stage framework capable of performing standard, promptable, and open-vocabulary visual relationship segmentation simultaneously. Our approach integrates human-centric (HOI segmentation) and generic VRS (Panoptic SGG) by adopting SAM (36] to unify different types of annotations into segmentation masks and using query-based Transformer architecture that outputs triplets in the format <sub ssubject, predicate, object>. The model enhances its interactive capabilities by accepting textual prompts as inputs. These prompts are converted into textual queries that assist the decoder in accurately identifying and localizing objects within the relationships. Additionally, we unify the labels from different datasets into a shared textual space, transforming classification into a process of matching with a set of textual features. Leveraging textual features from the CLIP model [64), we enable the effective matching of visual features with textual knowledge of novel concepts. This design

---

Figure 2: Examples of converting HOI detection boxes to masks. We filter out low-quality masks during training by computing loU between the mask and box.
inherently supports open-vocabulary and promptable relationship segmentation without pre-defining the number of object or predicate categories, facilitating dynamic and extensive adaptability.
Our FleVRS proposes unified framework that integrates standard, promptable, and open open-vocabulary VRS tasks into single system, as detailed in Tab. [T] providing greater flexibility compared to existing methods. It employs mask-based approach effectively manage various VRS tasks, enabling adaptation to different types of annotations, including HOI detection and panoptic SGG. Our architecture incorporates dynamic prompt handling, which supports both prompt-based and open-vocabulary settings, allowing our model to combine promptable queries with open vvcablocabylary capabilities to ground novel relational objects.
We evaluate our FleVRS on standard, promptable, and open- vocabulayy VRS tasks, i.e., HOI segmen- tation 4 18 and panoptic SGG [85). Crucially, we demonstrate competitive performance from three perspectives standard (40.5 vs. 39.1 mAP on HICO-DET [4),, promptable (56.8 vs. 33.5 sloU on
VRD [55],, and open- vocabulayy (31.7 vs. 25.6 mAP for "unseen object" on HICO-DET 4 visual relationship segmentation.
In summary, our main contributions are follows: 1) We introduce flexible one-stage framework capable of segmenting both human-centric and generic visual relationships across various datasets. 2) We present a promptable visual relationship learning framework that effectively utilizes diverse textual prompts to ground relationships. 3) We demonstrate competitive performance both standard close-set and open-vocabulary scenarios, showcasing the model's is strong generalization capabilities.
2 Related Work
Visual Relationship Detection (VRD) is split into two lines of works, including human-object interaction (HOI) detection 4 18 and panoptic scene graph generation (SGG) [77 83 They are defined as detecting triplets in the form of <aubject, subject object includes object box and category. HOI detection aims to detect human-centric visual relationships, while PSG focuses on generic object pairs' relationships. Previous works (I]
predicate, object> triplet, where
13 16
89 95 96 97 100 [144 [05] [06] usually train specialist models on single data source and tackle them separately. Departing from this traditional bifurcation. VniVRD [101] initiated the development of unified model for VRD. with subsequent efforts like [9]] 92] advancing relational understanding through large-scale language-image pre-training. Unlike the two-stage approach of (92), which performs object detection before decoding visual relationships, our method employs one-stage design that decodes objects and their relationships simultaneously. Crucially, our model extends beyond standard VRD capabilities to support promptable and open- vocabulary visual relationship segmentation, enhancing detailed scene comprehension.
32
35|
41
4T
47 57
81
Referring relationship and visual grounding. The most relevant work to is referring visual relationship introduced (38), where the model detects the subject and object depicting the structured relationship <subject, predi cate, object>. One-stage [38 75). two-stage [63 107 and three- stage (21) methods are proposed to localize the two entities' boxes iteratively based on the given structured prompt <subject, predi cate, object>. Unlike these methods, our approach allows for more flexible textual prompts without requiring the complete specification of the triplet. As shown in Fig. [ our model can handle queries that include a single item (e.g., predi cate) or a combination of two (e.g., predicate cate and object). Additionally, our method is capable of performing standard and open-vocabulary VRS. Visual grounding represents another related area, where models output
bounding boxes 7 8 20 29 40
52 83
object masks g
45
51 56
79
87 99 in response
to textual inputs. This process requires reasoning over the entities mentioned the text to identify the corresponding objects in the visual space. However, task fundamentally differs from this.
3

---

Decoder
Image Encoder
Input Image
Output Triplet
Qa
Latent queries
Textual queries
Subject mask
Object mask
Textual Encoder
<subject, object>
Relationship
Subject class Object class
Decoder
Promptable VRS
Predicate class
Figure 3: Overview of FleVRS. standard VRS, without textual queries, latent queries perform self- and cross-attention within the relationship decoder to output triplet for each query. For promptable VRS, the decoder additionally incorporates textual queries Q', concatenated with Q This setup similarly predicts triplets, each based on Q" outputs aligned with features from the optional textual prompt Q²
In FLeVRS, the promptable VRS task goes beyond mere identification; it involves outputting segmentation masks for both subject and object pairs along with categorizing their relationships. This capability is essential for understanding and interpreting complex relational dynamics.
Vision and language models Recent advancements in large-scale pre-trained vision- laniuae mod- els (VLM) 39 64 68
69,
82, 94] and multimodal large language models (MLLM) 2 5 76
78 88
have demonstrated impressive performance and generalization capabilities across a variety of vision and multimodal tasks [36 109 11). However, these models primarily focus on entity-level general- VRS receiving less attention. While recent efforts zero-shot HOI 80 92 often utilize CLIP [64] for category classification, their open-vocabulary input. Although current VLMs and MLLMs adept at grounding novel concepts from text, they require significant computational resources and additional visual models, and cannot directly generate comprehensive segmentation masks for subject and object pairs. In contrast, FleVRS provides lightweight solution that effectively supports various types open-vocabulary VRS, enabling category classification and the integration of novel concepts from prompts.
ization, with apen-vocabulasg
detection 47 61
capabilities lack the flexibility needed for prompt-drive
3 Method
3.1 Overview
Standard VRS. Given an image I, the goal of standard visual relationship segmentation (VRS) is to detect all the visual relationships of interest, either human-centris (i.e., HOI detection) or the generic ones (SGG), in terms triplets the form of <subject, predicate, object> (masks and object categories of subject and object, and the predicate category). The subject is always human in HOI detection, whereas it can be any type of object in SGG (may or may not be human). We consider the panoptic setting [85] of SGG, where model needs to generate a more comprehensive scene graph representation based on panoptic segmentation rather than rigid bounding boxes, providing a clear and precise grounding of objects. To produce fine-grained masks, we convert existing bounding box annotations from HOI detection datasets 4 [8] into segmentation masks using the foundation model SAM [36), as illustrated in Fig. 2 We employ filtering approach based on Intersection over Union (IoU) to filter out inaccurate masks. Details are in Appendix.
Promptable VRS. Our FLeVRS optionally accepts textual prompts as inputs, enabling users to specify visual relationships for promptable VRS. It accommodates three types of structured text prompts: single element (e.g., <?, predicate, ?>). any two elements (e.g., <subject, predicar cate, ?>, <subject, ?, object>), or all three elements. Consequently, the model outputs only the triplets that match the specified elements in the prompt, as depicted in the right column of Fig. [] Without textual prompts, functions as a standard VRS model, exhaustively generating all possible triplets, illustrated in the left part of Fig

---

Open-vocabulary VRS. In practice, it's essential for a VRS model to adapt to new concepts, including new categories of entities (i.e., subject and object), predicates. and their various combinations. Expanding these concept vocabularies to encompass a wider range is particularly challenging due to the vast potential combinations and the long-tail distribution of these categories. Thus, our goal is to equip the model to operate in an open-vocabulary setting, where it can effectively handle these diversities. it important to note that the above three capabilities are complementary; for example, the text prompts in promptable VRS can include novel object predicate categories.
To this end, we propose integrating the above three aspects into a single unified framework. Since these settings are complementary, a veneral-purpose model should be capable of performing various combinations of these three functions. Additionally, their inherent similarities make it more intuitive to consolidate them within a flexible, unified approach.
3.2 Model Architecture
we design a dual-query system for VRS model, illustrated in Fig. 3 Latent queries, a set of learned embeddings, generate triplets (which may be empty) to formulate output masks and relationship categories. For promptable VRS, textual queries derived from input prompts are incorporated. We employ an image encoder and a pixel decoder to extract visual features, coupled with relationship decoder that processes <subject, object> pairs and their interrelations. For open-vocabulary VRS, our approach shifts from traditional classification to matching strategy that aligns visual and textual features for both object and predicate categories, enhancing the model's adaptability to new concepts. Each component of this architecture is elaborated further below.
Inspired by the success of Transformer-based segmentation models 6 109
Image Encoder. Specifically, given the image I H.H it first fed into the image encoder Enc₁ to obtain multi-scale low-resolution features F = F. E RC
where the stride of
the feature map 8 E {4, 8, 16, 32). and CF is the number of channels.
Pixel Decoder. A Transformer- baasee pixel decoder Decp is used to upsample F and gradually generate high-resolution per-pins pixel embeddings P. is then passed to the relationship decoder DecR to compute cross-attention with query features.
Textual Encoder. When a text prompt is provided for promptable VRS, we use the textual encoder EncT to encode it into a set of textual queries Q E N.C, where N₂ is the number of tokens in the textual queries, and Cy denotes the channel number of query features. In practice, we use the textual encoder from CLIP [64] as EncT. The format of the text prompt can be single item (e.g. "/p=predicate</p>>)) two of them ("=s>>suhectt</>> <prrrts//>>>>>p///>>>>>>>>>>>>>> <p>>><>>>peppp <rpppppicatte/pp>>>
or all three of them, where "predicate" and "subject" denote category names of predi cate and subject, respectively. "
", "<o>" "<p>>" are used as separate tokens between subject, predi cate and object in the text prompt. We could use natural language the textual prompt instead of using structured format. However, collecting the textual VRD data is not trivial and we leave it as an extension of our model in future work.
Relationship Decoder. The relationship decoder DeCR, based on a Transformer decoder design, processes pixel decoder outputs P and latent queries Q" to generate all possible triplets for standard VRS. Inside, masked attention [6] utilizes masks from earlier layers for foreground information. Each Q" output feeds into five parallel heads: two mask heads for subject and object masks (M., Ma), two class heads for their categories (C., C.), and another class head for relationship prediction (C,) During training, Hungarian matching aligns predicted triplets with ground truth. For standard VRS inference, triplcts above a confidence threshold are considered final predictions. For promptable VRS, EncT transforms text prompts into textual queries Qf that are concatenated with Q" and input into DecR. This process, which uses self- and cross-attentie predicate, object> triplets, similar to standard VRS. An additional matching loss during training ensures the model predicts triplets as specified by the text prompt. During inference, we calculate similarity scores between the textual query feature (last token's 'ss feature of Q²) and the latent query outputs. We then select entities and relationships specified in the textual prompt from the top triplets for the final outputs.
mechanisms, generates <sub ject,,ct,
Matching with textual features. To enable open-vocabulary VRS, our FLEVRS uses the CLIP textual encoder (64] to match visual features with candidate textual features for object and predicate
5

---

categories. We convert these categories into textual features using prompt templates, such as "A photo of [predicate--ing for HOI segmentation and "A photo of something (predicate-ing] (something)" for panoptic SGG The model computes matching scores between predicted class embeddings and these textual features, allowing classification beyond the fixed vocabulary of the training set and facilitating open-vocabulary VRS. Textual prompis are similarly encoded, and their features are used to calculate similarity scores for promptable VRS inference.
3.3 Loss functions
We use Hungarian matching during training to find the matched triplets with ground truth ones. For standard VRS, we compute focal losses Lj L3 and dice losses L3. L2 on subject and object mask predictions, cross-entropy losses LE, Le. L2 on subject, object, and predicate category classifications, which can be written as
L =
Σ C (E(+,0)
Σ
C
Σ
(1)
jE(,,0)
kE(#,op))
where Ab. Ad and A are hyper-parametten for adjusting the weights of each loss. X X. X are different classification loss weights for subject, object, and predicate. For promptable VRS, we adopt an additional matching loss by between the matched triplet class embedding and the textual query feature (the last token feature of Qt), which is in the form of cross-entropy loss. The final training loss is written as
L=
Σ iE(x,0}
CJ + As
Σ jE(x,0}
C +
Σ kE(x,op)
Nr + Ag
where Xg controls the weight of Lgg L. depends on the text prompt. For example, given <subject, predicate>, there will not have Le and LE terms in Eq. ( with subject and predicate categories being given. See the appendix for the concrete values of loss weights.
Experiments
4.1
Experimental Settings
Datasets For HOI segmentation, we utilize two public benchmarks; HICO-DET 4 and V-COCO [8] To fit Our FleVRS, we use SAM (66 transform box annotations into masks and apply Non- Maximum Suppression (NMS) to remove overlapping masks with an IoU threshold greater than annotations from HICO-DET due to incomplete annotation, leaving 44,329 images (35,801 training, 8,528 testing) with 520 HOI classes from 80 objects and 116 actions V-COCO is built from COCO 149, comprising 10,396 images (5,400 training, 4,964 testing), featuring 80 objects and 29 actions, and includes 263 HOI classes, Both datasets align with COCO's object categories. For panoptic SGG, we use the PSG dataset (85) sourced from COCO and VG (37 intersections, containing 48, 48,749 images (46,572 training, 2,177 testing) with 133 objects and 56 predicates.
0.1. We omit no www.inneerctttteeeeeeeeeeeeeeeee
Data Structure for open-vocabulary HOI segmentation Following prior studies 3 23 we evaluate
HICO-DET under three scenarios: (1) Unseen Composition (UC), where some HOI classes are absent despite all object and verb categories being present; (2) Unseen Object (UO), where certain object classes and their corresponding HOI triplets are excluded from training: and (3) Unseen Verb (UV), where specific verb classes and their associated triplets are similarly omitted. In UC, the Rare First (RF-UC) approach targets tail HOI classes, while Non-rare First (NF-UC) focuses on head categories. Originally, UC included 120/480/600 categories for unseen/seen/full sets, which reduces to 115/405/520 after removing no _o_ vo_interaction annotations. For UO, we select 12 unseen objects from 80, resulting in 88/432 unseen/seen HOI categories.
Evaluation Metric For standard HOI segmentation, we convert the predicted masks to bounding boxes to compare with current methods, and follow the setting in [4] to use the mean Average Precision (mAP) for evaluation. We also turn the outputs of other methods into masks and report
I 'Omit "something" for spatial relationships. interrchangealle with "verb", "predicate".

---

(a) Text prompt: <?, object, predicate>
(b) Text prompt: <subject, predicate, ?>
flip, skateboard>
<?, wash, bus>
<person, hold, ?>
<person, pull, ?>
<?, ride, boat>
<?, hold, baseball glove>
<person, hold, ?>
<person, straddle, ?>
(c) Text prompt: <subject, ?, object>
<person, motorcycle> [ride, on]
<person, laptop> [hold, read]
<person, kite> [hold, carry]
<person, skis> [stand on, wear]
Figure 4: Qualitative results of promptable VRS on HICO-DET test set. We show visual- izations of subject and object masks and relationship category outputs, given three types text prompts. In (c), we show the predicted predicates in bold characters. Unseen objects and predicates denoted in red characters.
mask m.P for thorough comparison. An HOI triplet prediction is true positive if (1) both predicted human and object bounding boxes/masks have IoU larger than 0.5 w.t.f GT boxes/masks; (2) Both the predicted object and verb categories are correct. For HICO-DET,
evaluate the three different category sets: all 520 HOI categories (Full), 112 HOI categories (less than 10 training instances) (Rare), and the other 408 HOI categories (Non-Rare). For VCOCO, we report the role mAPs in two scenarios: (1) S1: 29 actions including 4 body motions; (2) S2: 25 actions without the no-object HOI categories. For standard panoptic SGG, following [85).
R@K K and mR@ K metrics, which calculate the triplet recall and mean recall for every predicate category, given the top triplets from the model. A successful recall requires both subject and object to have mask-based IoU larger than 0.5 compared to their GT masks, with the correct predicate classification in the triplet.
Implementation Details Following 6 109 we use 100 latent queries and 9 decoder layers in the relationship decoder. We adopt Focal-T/L 84 for the Image Encoder and Da ViT-B/L the pixel decoder. We use the textual encoder from CLIP encode input text prompt and subject, object, and predicate categories. During training, we set the input image to be 640 640, with batch size of 64. We optimize
network with AdamW W [54] with weight decay of 10 We train all models for 30 epochs with an initial learning rate 10 decreased by 10 times the 20th epoch. To improve training efficiency, all experiments, the parameters of the textual encoder are frozen except its logit scales. The loss weights λb, Aas d and Aged (superscript omitted) are set to 1,1,2, and 2. More details are in the appendix.
initialize Our FleVRS using the pre-trained weights from [109). For
4.2 Standard VRS
We evaluate our method on three benchmarks, i.e. HICO-DET VCOCO 18 for HOI segmenta- tion, and PSG 885 for the panoptic SGG.
HOI segmentation Since Our FleVRS leverages mask supervision, either converting mask results into bounding boxes transforming bounding boxes from previous methods' output into masks does not facilitate a completely equitable comparison. For the utmost fairness in comparison, we report both box mP and mask mAP from the above ways. As shown in Table 2 Our FLeVRS shows superior

---

Default (%)
Model
Backbone
box/mask mAP;
box/mask mAPR
box/mask m.P
Bottom-up methods
SCG 96
ResNct-50
31.3/
24.7 25.0
33.3 / 35.5
UPT 99
ResNet-101
32.6 34,6
28.6 29.4 29.4
33.8 36,1
STIP
100
ResNet-50
32.2 30.8
/28.
33.4 32.5
VIPLO 60
VIT-B
37.2 /3.1
35.57 37.8
37.8 / 39.7
Additional fraining with object detection data
UniVRD 101
ViT-L
37.47-
28.97
39.97-
PVIC 28
Swin-L
44.3/-
44.6/-
44.2 /-
RLIPv2
Swin-I.
45.1/48.6
45.6 44.3
43.2 49.88
Single-stete slage methods
HOTR 33
ResNet-50
25.1/26.5
17.37 18.5
27.4 29.0
QPIC
70
ResNet-101
29.9 30.5
23.0/23.1
31.7 33.1
CDN 95
ResNet-101
32.1/33.9
27.2/28.9
33.5 / 36.0
RLIP (V(VG-COOOO)
ResNet-50
32.8/34.
26.91
34.6 36.5
VLKT 47
ResNet-100
35.0
31.2/32.6
36.1 37.8
HRNet 48
EfffciennNerV2-X1
35.9
30.17-
38.3
MUREN
35
ResNet-50
32.9/35.4
28.7
34.1 37.6
Ours
Focal-L
38.1
33.0 34.9
39.5 42.4
Table 2: Quantitative results on the HICO-DET test set. We report both box and mask m. m.AP under the Default setting 4 containing the Full (F), Rare (R), and Non-Rare (N) sets. no no_interaction class is removed in mask mAP. The best score is highlighted in bold, and the second-best score is underscored. '-' means the model did not release weights and we cannot get the mask m.P. Due to space limit, we show the complete table with more models in the appendix.
Model
Backbone
APSt1
APS12
Bottom-up up methods
VSGNet 72
ResNet-152
51.8/ --
57.0/-
ACP
34
ResNet-152
53.2
-/-
ResNet-50
53.3
60.3/-
STIP 100
ResNet-50
66.0 / 66.2
70.7/700x
Additional training with object detection data
UnDDRD 101
ViT-L
65.1 /-
66.31-
PVIC 98
Swin-L
64.1/-
70.2 /-
RLIPv2
92
Swin-L
72.1/71.7
74.1/73.5
Single-stage methods
HOTR 33
ResNet-50
55.2/55.0
64.1
DIRV
EfficientDet-d33
56.1/-
-/-
CDN 95
ResNet-101
63.9/61.3
65.8/633 63.2
RLIP
91
ResNet-50
61.9/61.3
64.2/ 64.0
GEN-VLK
ResNet-101
63.6/61.8
64.0
ERNet
48
EfficientNerV2-XXL
64.2
-1-
Ours
Focal-1.
65.2 / 66.5 66.5 67.9
Table 3: Quantitative results on V-COCO. We report both box and mask mAP The best score highlighted in bold, and the second-best score is underscored. '-' means the model did not release weights and we cannot get the mask mPP. Due to space limit, we show complete table with more models in the appendix.
performance over current single-stage methods in terms of box and mask mAP P on HICO-DET. We also achieve competitive performance on VCOCO [18], as shown in Table The advantages of Our FLeVRS come from: (1) one-stage Transformer-baseee design with fine- fineggrined training supervision for VRS. With subject and object masks, the model has more accurate supervision, compared with box annotations that contain redundancy
[85). (2) good language-visua alignment with the large-scale pretrained model 644. Our FleVRS achieves competitive results without additional training on large-scale detection datasets [101]. Among one-stage HOI methods, our approach is simpler and able to tackle different datasets without modifications to the structure.
Panoptic SGG From Table 4 Our FleVRS can achieve competitive results in terms of R@50 and R@ 100 without elaborated designs for PSG, compared with most of previous work. Our FleVRS not superior HiLo [104), which is mainly due to the long-tail tail distribution of the dataset and the

---

Method
Backbone
R/mR@2
R/mR@50
R/mR @ 100
Adapted from SGG methods
IMP [8
VGG-16
17.9/
19.5 7.88
20.1 /
MOTIES 93
VGG-16
22.5 20.9/9.00
10.1
23.1 / 10.3
VCTree 71
VGG-16
21.7/99.8
23.3 / 10.2 20.0 18.4/6.52
23.7 10.3
Get 50
VGG-16
20.6/7.17
One-stage PSG methods
PSGTR [85]
ResNet-101
28.2/
32.1 / 20.3
35.3/21.3
PSGFormer 85
ResNet-101 101
18.0/ 14,2
20.1 / 18.3
19,0/198
Training with additional data
Hi 104
Swin-I.
40.6 29.7
48.7 /37.6
51,47 40.9
Ours
Focal-L
27.0/15.4
31.0/ 18.3
31.7/ 18.8 18.8
Table 4: Quantitative results on PSG. The best score is highlighted in bold, and the second-best score is underscored.
limitation of using CLIP to encode abstract relationships (e.g., entering, exiting). The model tends to predict high-frequency relationships and is hard to understand and predict low-frequency ones. Ablation study. We ablate Our FleVRS by testing different encoding strategies for relationships via the textual encoder in[7 Specifically, we compare encoding object and predi cate categories as <person, predicate, object> triplets or separately, associating the results with either triplet cross-entropy (CE) loss or disentangled CE loss. Results reveal that while HICO-DET benefits from the disentangled CE loss, allowing better generalization to novel concepts, VCOCO performs better with triplet CE loss due to the challenge distinguishing verbs without corresponding objects in various contexts (e.g.. differentiating "cat" in "a person eating apple" vs "a person cating"). Further experiments with various backbones demonstrate performance enhancements with larger models. Additionally, incorporating box head for supervision alongside mask supervision enhances performance, which is attributed to the masked attention mechanism inspired by (6) Exploring the potential synergies of training across multiple datasets, we find that while unified training improves VCOCO's performance due to its smaller size, HICO-DET and PSG show limited gains. This disparity likely due to the different predicate categories used in PSG compared to HICO-DET and VCOCO.
Comparison with previous works. UniVRD uses two-stage approach, where the model first detects independent objects and then decodes relationships between them, retrieving boxes from the initial detection stage. In contrast, where each query directly corresponds to a <subject, object, predicate> triplet. This tran- sition improves time efficiency from O(MxN) to O(K), where M is the number of subject boxes, N is the number of object boxes, and K is the number of interactive pairs. Our ap- proach also provides greater flexibility by learning unified representation that encompasses object detection, subject-object association, and relationship classification in a single model. In terms of training data, we much fewer train-
method employs a one-stage approach,
ing data (x50 less, without using VG (37 and Ob-
No subject
No object
Only predicate S-1oU O-JoU
jects365 (66)) and Our FleVRS with the Focal-L [84)
S-loU
0-1oU
backbone is much smaller than UniVRD [00 (164M
Comu-based methods
vs 640M) with LiT(ViT-H/14) we achieve comparable results(37.4 vs 38.1 on HICO-DET), While our method does not match RLIPv2 92 in performance, this is due to different design philosophies and goals. RLIPv2 a two-stage approach optimized for large-scale pre- training and relies on separately trained detectors. Our FleVRS, however, is not designed for pretraining and does not include separately trained detector. Our focus is on enhancing the flexibility of the VRS model without directly training on extensive curated data(x50 more, VG and Objects365). Thus, the differences in performance are attributed to the scale and design objectives. We further discuss the FLOPs and the number parameters of the backbone compared to previous works in the Appendix.
VRD 555
0,208
0.008
0.024
0.026
SSAS XS
0.335
0.363
0.334
0.365
Ours
0.568
0.364
0.556
0.366
Table 5:
Comparison of promptable VRD results with the baseline on VRD dataset (55).
4.3 Promptable VRS
We evaluate the ability of promptable VRS on the VRD dataset 55 to compare with 38 As in Table Our FLeVRS can locate entities given flexible text query inputs and performs better localizing

---

subjects and objects. Our FleVRS gets particular better results on localizing subjects (0.568 vs 0.335, 0.556 vs 0.334), which mainly because there are fewer categories in subjects compared with objects and lots of subjects are humans, making it easier to segment subjects. We further evaluate our promptable VRS approach on HICO-DET and PSG, as they contain rich relationship labels. Since there are no previous baselines, we show qualitative results in Fig. A object masks with the highest matching score for each example. We can see that the model is able to localize subject and object masks and predict their relationships given the structured textual prompt. We further perform postprocessing way to search triplets from standard VRS output, which serves another baseline to show the effectiveness of our method. Please refer to section E in the appendix for results and discussions of fair comparison.
We visualize the subject and
Difference with standard REC tasks The referring expression comprehension (REC) tasks on benchmarks like RefCOCO 300, RefCOCO+ [59], and RefCOCOg (90) are designed to detect objects based on free-form textual phrases, such as "a ball and cat" or "Two pandas lie climber."
In contrast, the promptable VRS task in our work focuses on detecting subject-object pairs within structured prompt format, such as <?, sit_on, bench> <person, ?, horse>, as illustrated in Fig. 1 of the main paper. Our FLeVRS is designed to encode and compute similarity scores for each of these elements separately, Our primary focus is relational object segmentation based single structured query, which differs significantly from the objectives of REC benchmarks.
4.4 Open-vocabulary VRS
We conduct open- spen-vocabulary experiments following the defined zero-shot HOI detection setting 23 25 26, (7) on HICO-DET. As shown in Table 66 Our FleVRS surpasses previous single-dataset methods across all settings, with its open-vocabulary capabilities stemming from the knowledge transferred from CLIP 64] GEN-VLKT [47] also leverages CLIP to facilitate ppeebvocayylary capabilities by encoding <person, predicate, object> a triplet and using for HOI category classification. In contrast, our approach separates the encoding of predicate and object, enhancing the model's generalization ability over novel concepts.
Method
Unseen
Seen
Full
VCOCO mask mAP; mask AP
HICO-DET
Raze
First Unseon Composition
R/mR 20
VCL
10.06
24.28
21.43
ATL
25
9.18
24.67
21.57
Different loszer
20
13.16
22.01
Disentangled CE loss
40.5
62.1
27.0 15.4
GEN
GEKT
4
21.36
32.91
30.56
Triplet CE loss
36,8
66.5
25.5/144 14.6
Ours
26.06
36.60
Disentangled
+ Triplet less
Non-rare First Unseen Composition
Different visual backbones
VCL
16.22
18.52
18.06
Focal Tiny
34,2
2
ATI
25
18.25
18.78
18.67
Focal Large
/
20
Different design choices
GEN VLKT
47
25.05
23.38
23.71
Box head only
33.0
62.0
Ours
26.62
31,17
30.17
Mask head only
40.5
66.5
27.0/554
Unseon Objece
Mask and box bead
41.2
67.0
26
Different training
ATL 25
5.05
14.69
13.08
Single source
40.5
66.5
27.0
10.51
MICODDET+CCKE
40.3
66.9
Ours
14.48
35.28
31.71
HICO-DET-VOOCO-PSG
27.6
Unseen kenb
GEN-VLKT 47
20.96
30.23
28.74
Table 7: Ablations of different loss types, backbones, design choices and training sets. We adopt the Focal- L backbone by default.
Ours
21.50
35.63
33.09
Table 6: Results of open-vocabulary HOI detection on HICO-DET.
5
Conclusion
In this work, we present a novel approach for visual relationship segmentation that integrates the three critical aspects of flexible VRS model: standard VRS, promptable querying, and -oca-vocyy capabilities. Our FleVRS demonstrates the ability to not only support HOI segmentation and panoptic SGG but also to do so in response to various textual prompts and across a spectrum of previously unseen objects and interactions. By harnessing the synergistic potential of textual and visual features, our model delivers promising experimental results existing benchmark datasets. We hope our work can serve as a solid stepping stone for pursuing more flexible visual relationship segmentation models.
10

---

References
[1] Abdelkarim, S., Agarwal, A., Achlioptas, P., Chen, J., Huang. J., Li, B., Church, K., and Elhoseiny, M. (2021). Exploring long tail visual relationship recognition with large vocabulary. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
[2] Bai, J., Bai, S., Yang. S., Wang. S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, J. (2023). Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. arXiv.
[3] Bansal, A., Rambhatla, S. S., Shrivastava, A., and Chellappa, R. (2020). Detecting human-object interactions via functional generalization. In AAAI.
[4] Chao, Y.- Liu, Y., Liu, X., Zeng, H., and Deng, J. (2018). Learning to detect human-object interactions. In WACV.
[5] Chen, K., Zhang, Z., Zeng, W., Zhang. R., Zhu, F., and Zhao, (2023). Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint arXiv:2306.151/
[6] Cheng, B., Misra, Schwing. A. G., Kirillov, A., and Girdhar, R. (2022). Masked-attentinnn mask transformer for universal image segmentation. In CVPR.
[7] Dai, X., Chen, Y., Xiao, B., Chen, D., Liu, M., Yuan, I.., and Zhang, L. (2021). Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7373-7382.
[8] Deng, J., Yang, Z., Chen, T., Zhou, W., and Li, H. (2021). Transvg: End-to-end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 1769-1779.
[9] Ding, Z., Wang, J., and Tu, Z. (2022). Open-vocabulaty panoptic segmentation with maskclip. arXiv preprint 2rXiv:2208.0 08984.
[10] Du H., Yu, X., and Zheng, L. (2020). Learning object relation graph and tentative policy for visual navigation. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK August 23-28, 2020, Proceedings, Part VII 16, pages 19-34. Springer.
[11] Fang, H.-S., Xie, Y., Shao, D., and Lu, C. (2021). Dirv: Dense interaction region voting for end-to-end human-object interaction detection. In Proceedings of the AAAI Conference Artificial Intelligence.
[12] Gao, C., Zou, Y., and Huang, J.-B. (2018). ican: Instance-centric attention network for human- object interaction detection. BMVC.
[13] Gao, C., Xu, J., Zou, Y., and Huang, J.-B. (2020). DRG: Dual relation graph for human-object interaction detection. In ECCV.
[14] Gkioxari, G., Girshick, R., Dollár, P., and He, K. (2018). Detecting and recognizing human- object interactions. In CVPR.
[15] Gu, J., Stefani, F.,, Wu, Q., Thomason, J., and Wang, X. (2022). -isionaadd-language navigation: A survey of tasks, methods, and future directions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 7606-7623.
[16] Gu, J., Wang, Y., Zhao, N., Xiong, W., Liu. Q., Zhang, Z.., Zhang, H., Zhang, J., Jung, H., and Wang, X. E. (2024a). Swapanything: Enabling arbitrary object swapping in personalized visual editing. arXiv preprint 4rXiv.240.005717.
[17] Gu, J., Fang, Y., Skorokhodov, I., Wonka, P., Du, X., Tulyakov, S., and Wang. X. E. (2024b). Via: A spatiotemporal video adaptation framework for global and local video editing. arXiv preprint arXiv:2406. 12831.
[18] Gupta, S. and Malik, J. (2015), Visual semantic role labeling. arXiv preprint arXiv: 1xxvv.1505.0447.

---

[19] Gupta. T., Schwing, A., and Hoiem, D. (2019). No-frills human-object interaction detection: Factorization, layout encodings, and training techniques. In ICCV.
[20] Han, Z., Zhu, F., Lao, Q., and Jiang, H. (2024). Zero-shot referring expression comprehension via structural similarity between images and captions. In Proceedings of the IEEE/CVT Conference on Computer Vision and Puttern Recognition, pages 14364-14374.
[21] He, C., Zhu, H., Gao, J., Chen, K., and Nevatia, R. (2020). Cparr: Category-baset proposal analysis for referring relationships. In CVPR workshops.
[22] Hong, Y., Rodriguez, C., Qi, Y., Wu, Q., and Gould, S. (2020). Language and visual entity relationship graph for agent navigation. Advances in Neural Information Processing Systems, 33, 7685-7696.
[23] Hou, Z., Peng, X., Qiao, Y., and Tao, D. (2020a). Visual compositional learning for human- object interaction detection. In ECCV.
[24] Hou, Z., Peng, X., Qiao, Y., and Tao, D. (2020b). Visual compositional learning for human- object interaction detection. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 584-600. Springer.
[25] Hou, Z., Yu, B., Qiao, Y., Peng, X., and Tao, D. (2021a). Affordance transfer learning for hujan-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages -95-504.
[26] Hou, Z., Yu, B., Qiao, Y., Peng, X., and Tao, D. (2021b). Detecting human-object interaction via fabricated compositional learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14646-14655.
[27] Hu, X., Lin, Y., Wang, S., Wu, Z., and Lv, K. (2023). Agent-centric relation graph for object visual navigation. IEEE Transactions on Circuits and Systems for Video Technology.
[28] Huang, Z.., Mo, X., and Lv, C. (2022). Multi-modal motion prediction with transformer-based neural network for autonomous driving. In 2022 International Conference on Robotics and Automation (ICRA), pages 2605-2611. IEEE
[29] Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I., and Carion, N. (2021). Mdetr- modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790.
[30] Kazemzadeh, S., Ordonez, V., Matten, M., and Berg, T. (2014). Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798.
[31] Kim, B., Choi, T., Kang, J., and Kim, J. (2020a). Uniondet: Union-level detector towards real-time human-object interaction detection. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XV 16, pages 498-514. Springer.
[32] Kim, B., Lee, J., Kang, J., Kim, E.S.,, and Kim, J. (2021a). HOTR: End-to-end human-object interaction detection with transformers. In CVPR.
[33] Kim, B., Lee, J., Kang, J., Kim, E.-S., and Kim, H. J. (2021b). Hotr: End-to-end human-object interaction detection with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 74-83.
[34] Kim, D.-J., Sun, X., Choi, J., Lin, S., and Kweon, I. S. (2020b). Detecting human-object interactions with action co-occurrence priors. In ECCV.
[35] Kim, S., Jung, D., and Cho, M. (2023). Relational context learning for human-object interaction detection. In CVPR.
[36] Kirillov, A., Mintun, F.,, Ravi, N., Mao, H., Rolland, C., Gustafson, I., Xiao, T., Whitehead, S., Berg, A. C.,, Lo, W.Y.,, et al. (2023). Segment anything. arXiv preprint arXiv:2344.02643
12

---

[37] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K. Kravitz, J., Chen, S., Kalantidis, Y., Li, L.-J., Shamma, D. A., et al. (2017). Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 32-73.
[38] Krishna, R., Chami, I., Bernstein, M., and Fei-Fei, L. (2018). Referring relationships. In CVPR.
[39] Li. J., Li, D., Xiong, C., and Hoi. S. (2022a). Blip: Bootstrapping language-iman image pre-training for unified
understanding and generation. In International conference on machine learning, pages 1 12900. PMLR.
[40] Li, L. H., Zhang, P.. Zhang, H., Yang, J., Li, C., Zhong, Y., Wang. L., Yuan, L., Zhang, L.. Hwang, J.-N., et al. (2022b). Grounded language-image pre-training. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10965-10975.
[41] Li, R., Zhang, S., Wan, B., and He, X. (2021). Bipartite graph network with adaptive message passing for unbiased scene graph generation. CVPR.
[42] Li, Y.-L., Zhou, S., Huang. X., Xu, L., Ma, Z., Fang, H.-S., Wang. Y., and Lu, C. (2019). Transferable interactiveness knowledge for human-object interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3585-3594.
[43] Li, Y.-I.,, Liu, X., Wu, X., Li, Y., and Lu, (2020). Hoi analysis: Integrating and decomposing human-object interaction. Advances in Neural Information Processing Systems, 33, 501 1 1-5022.
[44] Liang, C., Wu, Y., Zhou, T.. Wang, W., Yang, Z., Wei, Y., and Yang, Y. (2021). Rethinking cross-modal interaction from top-down perspective for referring video object segmentation. arXiv preprint arXiv:2 (00iv:2006.01/ 106.0106/.
[45] Liang, F., Wu, B., Dai, X., Li, K., Zhao, Y., Zhang. H., Zhang, P., Vajda, P., and Marculescu, D. (2023). Open-vocabulary semantic segmentation with adakead clip. In CVPR.
[46] Liao, Y., Liu, S., Wang, F., Chen, Y., Qian, C., and Feng, J. (2020). Ppdm: Parallel point detection and matching for real-time numan-objec interaction detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 482-490.
[47] Liao, Y., Zhang, A., Lu, M., Wang, Y., Li, X., and Liu, (2022). Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In CVPR.
[48] Lim, J., Baskaran, V. M., Lim, J. M.-Y., Wong, See, J., and Tistarelli, M. (2023). Ernet: An efficient and reliable human-object interaction detection network. IEEE Transactions on Image Processing.
[49] Lin, T.-Y,, Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects context. In ECCV.
[50] Lin, X., Ding, C., Zeng, J., and Tao, (2020). Gps-net: Graph property sensing network for scene graph generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3746-3753.
[51] Liu, C., Ding, H., and Jiang, X. (2023a). Gres: Generalized referring expression segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 23592-23601.
[52] Liu, S., Zeng, Z., Ren, T., Li, F., Zhang, H., Yang, J., Li, C., Yang, J., Su, H., Zhu, J., et al. (2023b). Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint 333.5:230..05499
[53] Liu, Y., Chen, Q., and Zisserman, A. (2020). Amplifying key cues for human-obheec-ineeeettio detection. In ECCV.
[54] Loshchilov. I. and Hutter, F. (2017). Decoupled weight decay regularization. arXiv preprint 1rX1v.1711.05101.
[55] Lu, C., Krishna, R., Bernstein, M., and Fei-Fei, L. (2016). Visual relationship detection with language priors. In ECCV.
13

---

[56] Lüddecke, T. and Ecker, A. (2022), Image segmentation using text and image prompts. In CVPR.
[57] Ma, S., Wang, Y., Wang. S., and Wei, Y. (2023). Fgahoi: Fine-grained anchors for human-object interaction detection. arXiv preprint wiXiv2231.000119
[58] Ma, X., J., Kochenderfer, J., Isele, D., and Fujimura, K. (2021). Reinforcement learning for autonomous driving with latent state inference and spatial-temporal relationships. In 202 IEEE International Conference on Robotics and Automation (ICRA), pages 6064-6071. IEEE.
[59] Mao, J., Huang. J., Toshev, A., Camburu, O., Yuille, A. L., and Murphy, K. (2016). Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11-00.
[60] Park, J., Park, J.-W., and Lee, J.-S. (2023). Viplo:
Vision transformer based pose-conditioned self-loop graph for human-object interaction detection. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pages 17152-17162.
[61] Peyre, J., Laptev, I., Schmid, C., and Sivic, J. (2019). Detecting unseen visual relations using analogies. In ICCV.
[62] Qi, S., Wang, W. Jia, B., Shen, J., and Zhu, S.-C. (2018). Learning buman-object interactions by graph parsing neural networks. In Proceedings of the European conference on computer vision (ECCV), pages 401-417.
[63] Raboh, M., Herzig, R., Berant, J., Chechik, G., and Globerson, A. (2020). Differentiable scene graphs. In WACV.
[64] Radford, A., Kim, W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In ICML.
[65] Shadlen, M. N., Britten, K. H., Newsome, W. T., and Movshon, J. A. (1996). A computational analysis of the relationship between neuronal and behavioral responses to visual motion. Journal of Neuroscience, 16(4), 1486-1510.
[66] Shao, S., Li, Z., Zhang, T., Peng, C., Yu, G., Zhang, X., Li, J., and Sun, J. (2019). Objects365:
A large-scale, high qugh-tuaitty dataset for object detection. In ICCV.
[67] Shic, F. and Scassellati, B. (2007). A behavioral analysis of computational models of visual attention. International journal of computer vision, 73, 159-177. 177.
[68] Singh, A., Hu, R., Goswami, V., Couairon, G., Galuba, W., Rohrbach, M., and Kiela, D. (2022). Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1 35638-15650
[69] Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., and Dai, J. (2019). VI-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv: arXiv:
[70] Tamura, M., Ohashi, H., and Yoshinaga, T. (2021). Qpie: Query-based pairwise human-object interaction detection with image-wiee wide contextual information. In CVPR.
[71] Tang, K., Zhang, H., Wu, B.. Luo, W., and Liu. W. (2019). Learning to compose dynamic tree structures for visual contexts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6619-6628.
[72] Ulutan, O., Iftekhar, A., and Manjunath. B. S. (2020). VSGNet: Spatial attention network for detecting human object interactions using graph convolutions. CVPR.
[73] Wan, B., Zhou, D., Liu, Y. Li, R., and He, X. (2019). Pose-aware multi-level feature network for human object interaction detection. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 9469-9478.
14

---

[74] Wang, H., Zheng, W.-s., and Yingbiao. L. (2020a). Contextual heterogeneous graph network for human-objer -bject interaction detection. In Computer Glasgow, UK, August 23-28, 2020, Proceedings, Part XVII 16, pages 248-2644 Springer.
2020: 16th European Conference,
[75] Wang, H., Du, Y. Zhang, Y., Li, S., and Zhang, L. (2022a). One-stage visual relationship refer- ring with transformers and adaptive message passing. IEEE Transactions on Image Processing.
[76] Wang, P., Yang. A., Men, R., Lin, J., Bai, S., Li, Z., Ma, J., Zhou, C., Zhou, J., and Yang, H. (2022b). Ofa: Unifying architectures, tasks, and modalities through a simple sequence--o--sequene learning framework. In International Conference on Machine Learning, pages 23318-23340. PMLR.
[77] Wang. T., Yang. T., Danelljan, M., Khan, F. S., Zhang, X., and Sun, J. (2020b). Learning human-object interaction detection using interaction points. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4116-412)
[78] Wang. W., Chen, Z., Chen, X., Wu, J., Zhu, X., Zeng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., et al. (2024). Visionllm: Large language model is also open-ended decoder for vision-centric tasks. Advances in Neural Information Processing Systems, 36.
[79] Wu, J., Jiang, Y., Sun, P.,
Yuan, Z., and Luo, P. (2022). Language as queries for referring video object segmentation. In Proceedings of the IEEE/CVÉ Conference on Computer Vision and Pattern Recognition, pages 4974-4984.
[80] Xu, B., Wong, Y., Li, J., Zhao, Q., and Kankanhalli, M. S. (2019). Learning to detect human- object interactions with knowledge. In CVPR.
[81] Xu, D., Zhu, Y., Choy, C. B., and Fei-Fei, L. (2017). Scene graph generation by iterative message passing. In Proceedings of the IEEE conference computer vision and patlern recognition, pages 5410-5419.
[82] Xu, J., De Mello, S., Liu, S., Byeon, W., Breuel, T., Kautz, J., and Wang. X. (2022). Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Puttern Recognition, pages 18134 18134-18144
[83] Yan, B., Jiang, Y., Wu, J., Wang, D., Luo, P., Yuan, Z., and Lu, H. (2023). Universal instance perception as object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325-15336
[84] Yang, J., Li, C., Dai, X., and Gao, J. (2022a). Focal modulation networks. NeurIPS.
[85] Yang, J., Ang, Y. Z., Guo, Z., Zhou, K., Zhang, W., and Liu, Z. (2022b). Panoptic scene graph generation. In ECCV.
[86] Yang, J., Peng, W., Li, X., Guo, Z., Chen, L., Li, B., Ma, Z., Zhou, K., Zhang, W., Loy, C. C., al. (2023). Panoptic video scene graph generation. In CVPR.
[87] Yi, M., Cui, Q. Wu, H., Yang, C., Yoshie, O., and Lu, H. (2023). simple framework for text-supervised semantic segmentation. In CVPR.
[88] You, H., Zhang, H.. Gan, Z., Du, X., Zhang, B., Wang. Z., Cao, L., Chang, S.-F. and Yang, Y. (2023). Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704,
[89] Yu, J., Chai, Y., Wang, Y., Hu, Y., and Wu, Q. (2021). Cogtree: Cognition tree loss for unbiased scene graph generation. In IJCAI.
[90] Yu, L.. Poirson, P., Yang. S., Berg, A. C., and Berg, T. L. (2016). Modeling context in referring expressions. In Computer AtiionEECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pages 69-85, Springer.
[91] Yuan, H., Jiang, J., Albanie, S., Feng. T., Huang, Z., Ni, D., and Tang, M. (2022). Rlip: Relational language-image pre-training for human-object interaction detection. In Advances in Neural Information Processing Systems.
15

---

[92] Yuan, H., Zhang, S.,. Wang, X., Albanie, S., Pan, Y., Feng, T., Jiang, J., Ni, D., Zhang, Y., and Zhao, D. (2023). Rlipv2: Fast scaling of relational language-iman image pre-training. In ICCV.
[93] Zellers, R., Yatskar, M., Thomson, S., and Choi, Y. (2018), Neural motifs: Scene graph parsing with global context. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5831-5840.
[94] Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., and Beyer, L. (2022). Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133.
[95] Zhang, A., Liao, Y., Liu, S., Lu, M., Wang, Y., Gao, C., and Li, X. (2021a). Mining the benefits of two-stage and one-stage hoi detection. In NeurIPS.
[96] Zhang, F. 7.,, Camphell, D., and Gould, S. (2021b). Spatially conditioned graphs for detecting human-object interactions. In ICCV.
[97] Zhang. F. Z., Campbell, D., and Gould, S. (2022a). Efficient two-stage detection of human-object interactions with a novel umary-pairwwy transformer. In CVPR.
[98] Zhang, F. Z., Yuan, Y., Campbell, D., Zhong. Z., and Gould, S. (2023a). Exploring predicate visual context in detecting of human-object interactions. In Proceedings of the IEEE/CVF International Conference on Computer Vision.
[99] Zhang, H., Li, F., Zou, X., Liu, S., Li, C., Yang, J., and Zhang, L. (2023b). A simple framework for open-vocabulary segmentation and detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1020-1031
[100] Zhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., and Chen, C.-W. (2022b). Exploring structure- aware transformer over interaction proposals for human-object interaction detection. In CVPR.
[101] Zhao, L., Yuan, L., Gong. B., Cui, Y., Schroff, F., Yang, M.-H., Adam, H., and Liu, (2023). Unified visual relationship detection with vision and language models. arXiv preprint arXiv:2303.0883
[102] Zhong, X., Ding, C., Qu, X., and Tao, D. (2020). Polysemy deciphering network for human- object interaction detection. In ECCV.
[103] Zhong, X., Qu, X., Ding, C., and Tao, D. (2021). Glance and gaze: Inferring action-aware points for one-stage numan-obbect interaction detection. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pages 13234-13243
[104] Zhou, Z., Shi, M., and Caesar, H. (2023a). Hilo: Exploiting high low frequency relations for unbiased panoptic scene graph generation. In ICCV.
[105] Zhou, Z., Shi, M., and Caesar, H. (2023b). Viprompt: Vision-language prompting for panoptic scene graph generation. arXiv:2311.1649)
[106] Zhu, F., Xie, Y., Xie, W., and Jiang, H. (2023). Diagnosing human-object interaction detectors. arXiv preprint arX.v:2308.08529
[107] Zhu, J. and Wang. H. (2021). Multiscale conditional relationship graph network for referring relationships in images. IEEE Transactions on Cognitive and Developmental Systems.
[108] Zou, C., Wang, B., Hu, Y., Liu, J., Wu, Q., Zhao, Y., Li, B., Zhang, C., Zhang, C., Wei, Y., el al. (2021). End-to-end human object interaction detection with hoi transformer. In CVPR.
[109] Zou, X., Dou, Z.- Yang. J., Gan, Z., Li, L., Li, C.,, Dai, X., Behl, H., Wang, J., Yuan, L.,, et al. (2023a). Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15116-15127.
[110] Zou, X., Yang, J., Zhang. H., Li, F., Li. L., Gao, J., and Lee, J. (2023b). Segment everything everywhere all at once. arXiv preprint 2xXv:2230..06718.
16

---

A Limitations and Future Work
Deploying our model in real-world scenarios requires specialized pretraining data for relationship un- derstanding, which is notably scarce. The lack of automated annotation pipelines and dependence on the CLIP model pose scalability challenges due specific resource requirements. Ideally, we aim for a single ceneral-purposs framework that can be trained on multiple datasets and enhance performance across various tasks and benchmarks. However, achieving this remains a challenge with our current model. We leave the exploration of how to synergize different datasets and develop effective training strategies to future work. While integrating free-form text inputs more natural as large language models evolve, it necessitates additional preprocessing to align with our framework. Furthermore, the absence of comparable methods for promptable VRS makes complete fair benchmarking difficult.
B Model Structure Details
We use Focal T/L. 84 network as the image encoder Enc, Given the image E TR. Hx x3 x3 we pass it to Enc, and obtain multi-scale features of different strides and channals F [F,|s 4,8, 32). where s the stride.
Then, the pixel decoder Decp gradually upsample F to generate high-resolution per-pixel embeddings P (P i 2,2.3, 4]. where i is the layer number and different Pts have the same channel number but different resolutions. P will then input to DecR.
Under standard VRS, DecR takes latent queries Q" and P as inputs. Under promptable VRS, use the textual encoder EncT to encode the textual prompt into a set of textual queries Qf and concatenate Q' with the latent queries Q", and input them to DeCR. Inside DeCR, cross- and self-attention are computed among queries and per-pixel emheddings, where masked attention 6 is adopted to enhance the foreground regions of predicted masks.
On top of latent queries output Q E NC,, (N₂, is the number of latent queries, C, is the channel number), there are five heads, producing predictions in parallel. They are two mask heads fM, (-), fas, (.) for predicting subject and object masks (M,, Ma), and two class heads 9c, (-), 9c, (-) for predicting their object categories (C., Co). Another class head gcp is used to predict relationships Cp for this <subject, object> pair. Detailed operations can be written as
M, = Up [P4 SM, (x))
(3)
M. Up [P4 (Ox)
C, = T, 9c, (H))
(5)
C. T, IC. Qx)
(6)
C = Tp 9c, Q%),
(7)
where Up[-] denotes the upsampling operation, T., To and Tp denote candidate textual features of subject, object and predicate categories that are encoded by CLIP [64). The mask embeddings fas, (QX) and far, (Q)) compute the dot products with the last layer's ' per-pixel embedding P4. respectively, and upsample to the original resolution as final mask predictions.
Training. We employ Hungarian matching to align predicted triplets with ground truth, calculating mask and category classification losses on these matches. For promptable VRS, we introduce a matching loss to assess the similarity between the matched triplet embedding and the textual prompt's feature, formulated as a cross-cntropy loss. The triplet embedding combines class embeddings from class heads. For the textual prompt's feature, we use the last token's 's feature from textual queries Q*. For instance, with a textual prompt like <subject, predicate, ?>, where the subject and predicate are specified, the similarity measurement utilizes the summation of their class embeddings Ic, (Qx) + 9c, (Q.).
Inference. Under standard VRS, we compute the confidence score of each triplet, which comes from the product of subject, object, and predicate classification scores. We take top k, (k, 100) triplets to compute mean average precision for HOI segmentation and mean recall for panoptic SGG. Under promptable VRS, we compute similarities between the textual prompt's 's feature and triplet embeddings and choose ky (ky- 10) the final predicted triplets.
17

---

(a) Text prompt: <?, object, predicate>
(b) Text prompt: <subject, predicate, ?>
< walk, bicycle>
<?, load, bus>
<person, load,
<person, sit on,
<?, sit on, coach>
<?, jump, skateboard>
<person, lie on, ?>
<person, straddle, ?>
(c) Text prompt: <subject, ?, object>
<person, dining table><<erson, 2, skateboard> (eat_at, sit_at] [jump, ride]
<person, 2, hotdog> [eat, hold]
<person, ?, knife> [hold, wield]
Figure 5: Qualitative results of promptable and open-vocabulary VRS on HICO-DET 4 test set. We show visualizations the predicted triplet with the highest matching score, including subject, object masks, and predicted predicate categories. There are three types of textual prompts shown in (a), (b), and (c), with unseen concepts in the rightmost columns. In (c), we show the predicted predicates in bold characters. Unseen objects and predicates are denoted in red characters. Note that the subject is always "person" in HICO-DET.
C Implementation details
We set the input image size to be 640 x 640, with batch size as 64. The model is optimized with AdamW [54) with weight decay of 10-4 We set N, 100 and N, 200 for Focal-T and Focal-L backbones, respectively, and Cg = 512. The structure of pixel decoder Decp is Transformer encoder with 6 encoder layers and heads. The structure of relationship decoder DeCR. is a Transformer decoder with 9 decoder layers.
Standard HOI segmentation The model only takes the image input without textual prompts. Since the subject class is always "person" in HOI segmentation, we omit the subject class head. The model trained with 30 epochs, with initial learning rate of 10 (10 for the image encoder) decreased by 10 times at the 20th epoch. The loss weights λs da. X and Xe are set to be 2, 1, 1, 2.
Promptable HOI segmentation To enable promptable HOI segmentation, build three types of tex- tual prompts: 1)
3) "<S>>prron<//>><> >/ppsoo<//>>>oo <o>o///>>>>/////////// ppreecpppapaae////p>>>>> 2) "<p>>predccat=</p <o>ojp>>>>>>/////>/>>>>>>>> We evaluate the model HICO-DET 4 since it contains richer human-object interactions than VCOCO [18). During training, we randomly sample various types of text prompts and simultaneously train different objectives using distinct loss terms. To prevent the model from learning shortcuts, we select one ground truth triplet per training image and pair it with a randomly chosen textual prompt type. This approach ensures balanced distribution of labeled training data for promptable VRS across different prompt types.
Standard panoptic SGG We use the subject class head to predict the subject category and the model does not have textual prompts as inputs. The model trained with 60 epochs, with an initial learning rate of 10 (10-5 for the image encoder) decreased by 10 times at the 40th epoch. The loss weights Xo Ad Ae X and X are set to be 2, 1, 1, 1,
Promptable panoptic SGG Similar to promptable HOI segmentation, there are three types of textual "<p2predicate<//p> object</o>"; 3) *=>=bbbert
>>>>>ojjjttt Similarly, during training, randomly
prompts: 1)"H "
> subjjectdde-----pppedaeeeeeeeee
18

---

(a) Text prompt: <?, object, predicate>
(b) Text prompt: <subject, predicate, ?>
<?, standing on, road>
<?, on, floor>
<truck, on,
<person, hold,
??, driving on, road>
2, pavement>
<person, ride,
<car, driving on,
(c) Text prompt: <subject, ?, object>
<bottle, cabinett>> [on]
<coach, floor> [over]
<person, surfboard> [in, lie carry, touch]
<teddy bear, chair> [over, beside, on]
Figure 6: Qualitative results of promptable and open-vocabulary VRS on PSG [85] test set. We show visualizations the predicted triplet with the highest matching score, including subject, object masks, and predicted predicate categories. There are three types of textual prompts shown in (a), (b), and (c), with unseen concepts in the rightmost columns. In (c), we show the predicted predicates in bold characters. Unseen objects and predicates denoted in red characters.
sample different types text prompts, and different objectives are trained simultaneously with different loss terms. We also keep balanced distribution of labeled training data for panoptic SGG. We set the weight of grounding loss Xg to be 2, while other weights are the same as standard panoptic SGG.
Open-vocabulate promptable VRS We adopt the zero-shot setting from 47 for open-vocabulary HOI segmentation. To streamline approach, integrate the oocn-vocabulary promptable setting within the broader context of vocc-vocabllary VRS. In this setting, Sopen-vocabulay handling both seen and unseen categories in the input textual prompts. We randomly exclude object and predicate categories during training and assess our model on these as well as on seen categories. Given the absence of existing benchmarks for this specific challenge, present qualitative results demonstrating our model's proficiency in open-vocabulary promptable VRS.
refers to
D Qualitative results of promptable and open-vocabulary VRS
We show qualitative results of promptable and open-vocabulary VRS on HOI segmentation and panoptic SGG by giving the model different types of structured textual prompts. For simplicity, show examples of omitting only one component of the triplet. We can see that our model is able to localize the correct subject and object and complement the missing element corresponding given textual prompt, e.g. <person, ?, dining table>> in Fig. 5) and <truck, on, ?> in Fig. 6(b). The model can also predict multiple interactions for the same subject-object pair, shown Fig [[ c) and Fig. c).. We further trained two versions by removing unseen objects and unseen predicates, respectively. We show that our model can detect novel objects and predicates by feeding unseen concepts in textual prompts, as in the rightmost columns of Fig. and Fig. 6 Fig. b shows the model outputs multiple instances in one subject mask due to similar patterns occurring in the training set. Note that the flexible VRD task is more difficult on the PSG [85 dataset due to its complexity of scenes, while we make the first attempt and our model still able show promising grounding results.
the
19

---

Default (%)
Model
Backbone
box/mask mAPF box/mask mAPR box/mask mAPN
Bottom-up methods
InteractNet 14
ResNet-50
9.97-
7.21-
10.8 /-
ICAN 12
ResNet-50
14.8/
10.5
16.2 /-
No-Frills 19
ResNet-152
17.2
12.21
18.7
DRG 13
ResNet-50
24.5/
19.5
26.0/
VSGNet 72
ResNet-152
19.8/-
16.1
20.9/
FCMNet 53
ResNet-50
20.4/-
17.3/-
21.6/-
IDN 43
ResNet-50
23.4/
22.51-
23.6/-
ATL
ResNet-101
23.87
17.41
25.71-
SCG 96
ResNet-50
31.3//1.3
24.7/25.0
3 35.5
UPT 197
ResNer- 101
32,6/34.9
28.6/
33.8 36.1
STIP
100
ResNet-50
32.2/30.8
28.2 128.6
33.4 32.5
ViPLO
660
ViT-B
37.2/39.1
3
37.8/ 39.7
Additional training with object detection data
UniVRD [0]
VIT-L
37,41-
28.97
39.97
PVIC
58
Swin-I.
44.31-
44.6
44.2 /-
R
Swin-L
45.1
45.6/44.
43.2
Single-stage methods
DIRV 11
EfficientDet-d3
21.8
16.4
23.4/
PPDM-Hourglas
46
DLA-34
21.9/
13,9
24.3 /
HOL-Transformer
108
ResNet-101
26.6 /-
19.2
28.8 -
GGNet 103
Hourglass-10s
29.2
22.1 /-
30.8 /
HOTR 63
ResNet-50
26.1
17.3 18.5
27.4/
QPIC 70
ResNet-101
29.9/30.
23.0/23.1
31.7
CDN 995
ResNet-101
32.1
27.2/
33.5 36.0
RLIP (VG+COCO
ResNet-50
32.8/34.4
26.9/27.7
34.6/ 36.5
GEN-VLKT
ResNet-100 101
35.0
31.2/32.6
36.1 37..8
FRNet
4X
EfffcientNarV2-X11
35.9/-
30.1/-
38.3
MUREN 35
ResNet-50
32.9/35.4
28.7/30.1
Ours
Focal-L
38.1 40.5
33.0/ 34.9
39.5 / 42.4
Table 8: Quantitative results on the HICO-DET test set. We report both box and mask m A under the Default setting [4] containing the Full (F), Rare (R), and Non-Rare (N) sets. class is removed in mask mAP. The best score is highlighted in bold, and the second-best score underscored. *_' means the model did not release weights and we cannot get the mask m.PP.
Interaction
E
Quantitative results of standard VRS
Due to the large number of works on HOI detection, we show the complete comparison with previous methods in Fig. [ and Fig. 9 Our model achieves competitive results on both datasets, especially compared with other single-stage methods. From Table 9 MUREN (35) gets the best result on VCOCO (68.8 vs. 65.2, 68.2 2 vs. 66.5), but cannot achieve similarly strong result HICO-DET (32.9 vs. 38.1, 35.4 vs. 40.5), where the verb categories are more complicated.
Fair Comparison. Since existing models use bounding box annotations to train and evaluate m.PP, we ensure fair comparisons by converting our model's output masks into bounding boxes compute box m.AP. Additionally, we apply released weights from previous methods, transform their output boxes into segmentation masks using SAM (36], and report mask m.P.. In both metrics, our model demonstrates superior performance.
We further train the existing HOI detectors CDN (55. STIP, GEN-VLKT the same SAM generated data used our paper, which leads to worse accuracy on HICO-DET, as in Tab. TO] Thus, the major performance improvements of work are due to both the SAM-labeled data and our architectural design.
At the same time, we also train model with bounding boxes only, where we get decreased accuracy (mAP of 30.7 vs 36.3). We attribute it to the network architecture derived from Mask2Former [ which is mainly designed for pixel-wise segmentation tasks.
FLOPs and the number parameters of the backbone compared to previous works. in Table 2 and 3 of the main paper, we have done extensive comparisons with previous methods, including backbones on ResNet-50/101/155, EfficientNet, Hourglass, Swin Transformers, and LiT architectures. For previous methods that utilize ResNet backbone for HOI detection and PSG, our comparison
20

---

Model
Backbone
Ap
Ap
Bottom-up methods
InteractNet
RexNet-50
40.07-
GPNN
ResNet-50
44.0/-
ICAN
ResNet-50
45.3/-
52.41-
TIN 4
ResNet-50
47.81-
54.2/-
DRG
L3
ResNet-50
51.0/-
IP-Net 77
ResNet-50
51.0/-
-1-
VSGNet
72
ResNet-152
51.8/
57.0/-
PMFNet
73
ResNet-50
52.0/-
-/-
PD-Net
102
ResNet-50
52.67-
-/-
CHIGNet
ResNet-50
$2.7/-
FCMNet
ResNet-50
53.1/-
ACP 34
ResNet-112
53.2/-
IDN 43
ResNet-50
53.3/-
60.3 /-
STIP 100
ResNet-50
66.0/66.2
70.7 70.5
Additional Draining with object detection data
VCL
24
ResNet-101
48.31-
-1-
SCG
ResNet-50
54.2149.2
60.9/53.4
UPT
ResNet-101
61.3/60.3
67.1
UniVRD
VIT-I.
65.11-
66.3
PViC
98
Swin-L
64.1
70.2
RLIPv2
Swin-L
72.1
74.1 /7..5
Single-stage methods
UnionDet
Ki]
ResNet-50
47.5/-
56.2
HOI-Transformer
108
ResNet-101
52.9/-
-/-
GGNet [033
Hourglass-100 104
$4.71-
-1-
HOTR
33
ResNet-50
55.2/55.0
64.4 64.11
DIRV
EfficientDet.d3
56.1/-
-1-
QPIC
ResNet-101
58.31-
60.71
CDN
ResNet-101
63.966133
63.8/63.2
RLIP 91
ResNet-50
61.9/613
64.2/64A 64.0
GEN-VLKT'
47
ResNet-101
63.6/6188 65.9/64.0
ERNet
48
EfficientNetV2-XX
64.21-
-/-
MUREN 35
ResNet-50
68.8 /68.2
71.0
Ours
Focal-L
65.2 66.5
66.5
67.99
Table 9: Quantitative results V-COCO. We report both box and mask M...hhhe best score is highlighted in bold, and the second-best score underscored. means the model did not release weights and cannot get the mask mAP.
Model CDN
Trained with original boxes Trained with SAM masks
31.4
28.3
STIP
32.2
29.7
VLK-V
35.6
32.1
Results box mAP on HICO-DET test We train existing HOI detectors with Table 10:
mask head, by using the masks we generated through SAM.
includes VSGNet, ACP, No-Frills, which ResNet-152. To the best of knowledge, larger ResNet, such as ResNet-200, and ResNet-269, are not used in previous methods on related tasks. ResNet, have included the largest model ResNet-152, which has 65M parameters and 15 GFLOPs. Other baselines are not using the ResNet backbone, for example, UniVRD is using LiT(ViT-H/14) backbone. has 632M parameters and 162 GFLOPs, lot more than our 19r198M parameters and 15.6 GFLOPs, but still performs worse than our model.
Fair comparison of promptable VRS
Postprocessing of standard VRS outputs. Since no existing models share the same settings as promptable VRS, we create baseline for fair comparison. Typically, promptable VRS can be addressed by filtering standard VRS outputs. We post-process outputs from our standard VRS model to extract the desired triplets and compared their mAP with those from promptable VRS. The post-processed results yield lower mAP (15.7 vs. 26.8), primarily because the selected triplets often have lower confidence scores. Additionally, the post-processing approach is slower, taking seconds compared to 5 seconds for directly prompting the model to retrieve the desired triplet.
21

---

Grounding ability compared with prompt-based vision-language models. Although promptable VRS is similar to vision-language models like GLIP 440] and MDETR (29) in grounding capabilities, it has distinct objectives. Unlike these models, which focus on entities, promptable VRS outputs triplets, making direct comparisons infeasible. Previous models are not equipped to handle the promptable relationship understanding task directly. To explore this, we modify our structural design to incorporate multiple text prompts as inputs, which are individually processed with their matching scores aggregated for classification. This experimental setup, however, results in reduced performance, increased inference time (26s vs. 58), and higher GPU memory usage (5G vs. 3G). Thus, we argue that the proposed structure is suitable for tackling promptable VRS.
G Masks generated by SAM
Clarifications of choosing segmentation masks We firstly illustrate the importance of choosing segmentation masks over hoxes in Fig. Traditional bounding boxes often include overlapping and ambiguous information, leading redundancy. Segmentation masks, by accurately delineating object boundaries, provide a more precise and clear representation, reducing such redundancy, which is also illustrated in [85] and (86). Besides, segmentation masks provide enhanced visual understanding and comprehensive contextual analysis. Additionally, object detection models often struggle to precisely extract foreground objects, which is why they are typically combined with segmentation models like SAM for -ne-grieeed image tasks. Our model, however, presents a unified model that can localize both subjects and objects, along with their corresponding segmentation masks.
Figure 7: Illustration of the importance of using masks instead of bounding boxes. We show examples where one object is occluded by other objects. We show both bounding box annotations and masks generated with SAM, where only the masks can correctly locate the pure object.
Noise handling in using masks generated by SAM. To address potential noise and inaccuracies in masks generated by SAM, we employ filtering approach based on Intersection over Union (IoU). We compute the IoU between the generated masks and the original box annotations. Masks with an IoU score below a threshold of 0.2 are considered to have significant deviations from the ground truth and are filtered out. This threshold chosen to balance the trade-off between including sufficient mask data and excluding those with substantial inaccuracies. The chosen IoU threshold helps ensure that only masks with a reasonable overlap with the ground truth annotations are retained. This threshold is set based empirical evaluation and aims to minimize the impact of masks that are too noisy incorrect, while still retaining as much useful data as possible. After using this strategy, we conduct analysis on 200 samples. We tested various thresholds and found this gets the best balance between denoising and data retaining(95% valid data retraining).
More visualizations of generated masks. We have included additional visualizations in Fig. S to illustrate the fine-grained masks generated from the bounding box annotations of existing HOT detection datasets. These visualizations indicate that converting to masks significantly reduces the redundancy in the box annotations. Additionally, as shown in Fig. g (d), filtering with loU helps climinate low-quality masks.
22

---

(a)
(b)
(c)
(d)
Figure 8: Samples of fine-grained masks generated by converting existing bounding box annofa- tions with SAM. Samples are chosen from the HICO-DET dataset. Green boxes are original box annotations. Duplicated boxes are suppressed after converting the mask, as shown in (a). There also failure cases where no masks are generated with the given box annotations, as in (d).
23

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
In the abstract and section introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
In section 5.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
We do not have theoretical result in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
We provide implementation details section 4 and supplementary section B.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
cessiinnooccyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /pipp/.cc/pullc///uuii wwwiicccguie//eeeeeeeeeii CoodeSubeiiie ssi ssionnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: In section and supplementary section B.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
26

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: In section and supplementary.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: http::::////ips..eeeee http://s...............ccccceeaa
Answer: [Yes]
Justification:
We have made sure.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Do not have societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
27

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Do not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Citations are complete.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworeewwthhhe....c. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
28

---

Answer:
[NA]
Justification: No new assets released.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "MFU9otTpZs", "reviewer": "Reviewer_LLLd", "summary": "This paper propose a model to handle multiple visual relationship tasks, like HOI detection and Scene Graph Generation.\n\nThe proposed model is based on vision-language models similar to CLIP.\n\nIt handles different formulations like standard close-set, open-vocabulary, and prompted setting.", "questions": "- What's the essential difference between this model and UniVRD?\n- What's the necessity for segmentation rather than prediction?", "limitations": "Limitations are discussed.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Unified model for HOI and SGG.\n- Multiple inference setting supported, like open-vocabulary, standard close-set, and prompted.\n- The writing is smooth, and the demonstration is generally OK.", "weaknesses": "- Why not detection, rather than segmentation? What's the necessity for segmentation-style output rather than traditional detection style tasks? I would like more clarification on this.\n- Also, (if not using the segmentation setting,) this method could be compared with more methods. Currently, the compared methods seems still not sufficient.\n- This general contribution and pipeline is similar to UniVRD, except (1) this model is mask-based rather than box-based, and (2) a prompted setting is further supported. Thus, the contribution seems incremental.\n- The performance is basically the same as UniVRD and is not as good as RLIPv2.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful feedback. Please refer to “The General Response to Reviewers” for the reply to the issue of the difference with previous methods and refer to the PDF for illustration of the importance of using segmentation masks. We respond below to your other comments and questions.\n \n1. **Advantages of segmentation over detection.** \n\n   - Precision and Reduced Redundancy: Traditional bounding boxes often include overlapping and ambiguous information, leading to redundancy. Segmentation masks, by accurately delineating object boundaries, provide a more precise and clear representation, reducing such redundancy, which is also illustrated in [1] and [2].\n    - Enhanced Visual Understanding: Segmentation allows for a more detailed analysis of visual relationships, crucial for tasks like panoptic scene graph generation and human-object interaction detection. It enables the model to better distinguish between object parts and backgrounds, leading to improved performance in complex scene understanding. As in Table 7 of the main paper, using box head leads to performance drop in 7.5 mAP on HICO-DET and 4.5 mAP on VCOCO.\n    - Comprehensive Contextual Analysis: Bounding boxes typically miss important contextual elements, such as background categories like roads, sky, and trees, as shown in the standard PSG task in Figure 1 of the main paper.  Segmentation captures these elements, offering a more complete understanding of the scene, which is vital for tasks like image and video editing.\n    - Unified Model: Additionally, object detection models often struggle to precisely extract foreground objects, which is why they are typically combined with segmentation models like SAM for fine-grained image tasks. Our approach, however, presents a unified model that can localize both subjects and objects, along with their corresponding segmentation masks. We show visualization to illustrate this point in the attached PDF.\n\n     We hope this clarifies the advantages of using segmentation masks in our approach.\n\n2. **Could Compare with more methods if not using the segmentation setting.**\n\n    Thank you for the valuable feedback! We actually conducted comparisons without the segmentation setting, as detailed in the paper. Besides computing mask mAP in segmentation setting, we also conducted box mAP in non-segmentation setting. We did this by converting our segmentation masks into bounding boxes and transforming the box outputs of previous methods into masks. In Table 2 and 3 in the main paper, we show comparison with 13 previous methods, including one-stage methods, two-stage methods, with different backbones(ResNet, Swin, ViT) and training data, and in Table 8 and 9 in the appendix, we show full comparison with over 25 methods.\n\n[1] Yang, Jingkang, et al. \"Panoptic scene graph generation.\" ECCV 2022.\n\n[2] Yang, Jingkang, et al. \"Panoptic video scene graph generation.\" CVPR 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cakVhsyqBX", "reviewer": "Reviewer_W5TQ", "summary": "This work proposes an approach for visual relationship segmentation that integrates the three aspects of a VRS model: standard VRS, promptable querying, and open-vocabulary capabilities. The idea of the article is very good, but the performance seems to be lacking.", "questions": "see weakness", "limitations": "see weakness", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Enhancing HOI from the perspective of object segmentation is an interesting and promising idea.", "weaknesses": "1-Previous work mostly uses ResNet series backbones, while the authors use a Focal-L Backbone. How do the FLOPs and parameter counts of this backbone compare to ResNet? I am concerned there may be an unfair comparison.\n\n2-The performance of the proposed method does not seem to be particularly superior, which is a significant drawback. For example, in Tables 2 and 3, UniVRD, PViC, and RLIPv2 significantly outperform the proposed method in terms of box mAP metrics. Can the authors analyze this situation?\n\n3-In line 208, the authors mention using SAM to generate masks. How do they handle the noise in these masks? Some masks might significantly deviate from the ground truth.\n\n4-There are some typos, such as in line 25: \"textttperson\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful feedback. Please refer to “The General Response to Reviewers” for the reply to comparison of our backbone with ResNet's and refer to the PDF for visualizations of masks generated by SAM. We respond below to your other comments and questions. \n\n1. **Performance seems not to be particularly superior. Can the authors analyze this situation?**\n\n\n    (1) ***Training Data and Model Efficiency***: Our approach is designed to be more data-efficient and requires fewer training data compared to methods like RLIPv2 and UniVRD, which use extensive datasets. For example, they used both VG (108,077 images) and Objects365 (more than 2,000,000 images), while our model only used single data source, e.g. HICO-DET (47,776 images), which is x50 smaller. The fact that our model does not rely on these large datasets impacts the box mAP performance in direct comparisons.\n\n    (2) ***Model Complexity and Two-Stage Approach***: UniVRD, PViC, and RLIPv2 employ two-stage methods and fine-tune object detectors using extra datasets, which contribute to their higher box mAP scores. In contrast, our model uses a one-stage approach without separate object detection fine-tuning. This simple design choice, while impacting box mAP, aligns with our goal of creating a more flexible and data-efficient model.\n\n    (3) ***Model Capacity***: Our model, utilizing the Focal-L backbone, is significantly smaller in terms of capacity compared to UniVRD. For instance, the Focal-L backbone has 198M parameters compared to 632M in UniVRD with the LiT (ViT-H/14) model. Despite this difference, we achieved better results in terms of overall performance metrics, such as 37.4 vs 38.1 on HICO-DET, which highlights the efficiency and effectiveness of our approach despite its smaller size.\n\n    To conclude, our model prioritizes data efficiency and simplicity over sheer complexity, utilizing fewer training data and a one-stage approach without additional fine-tuning on extra datasets. Despite having a significantly smaller model capacity, our approach still achieves comparable overall performance, though at the cost of slightly lower box mAP scores. This trade-off reflects our focus on creating a more flexible and efficient model rather than optimizing for specific metrics like box mAP.\n\n\n2. **Noise handling in using masks generated by SAM.**\n\n    To address potential noise and inaccuracies in masks generated by SAM, we employ a filtering approach based on Intersection over Union (IoU). \n    - IoU-Based Filtering: We compute the IoU between the generated masks and the original box annotations. Masks with an IoU score below a threshold of 0.2 are considered to have significant deviations from the ground truth and are filtered out. This threshold is chosen to balance the trade-off between including sufficient mask data and excluding those with substantial inaccuracies.\n    - Rationale for IoU Threshold: The chosen IoU threshold helps ensure that only masks with a reasonable overlap with the ground truth annotations are retained. This threshold is set based on empirical evaluation and aims to minimize the impact of masks that are too noisy or incorrect, while still retaining as much useful data as possible. Please refer to the attached PDF for visualizations of generated masks from bounding box annotations. After using this strategy, we conduct analysis on 200 samples. We tested various thresholds and found this gets the best balance between denoising and data retaining(95% valid data retraining).\n\n\n3. **Typos.**\n\n    Thank you for pointing out the typo. We will ensure that the final version of the manuscript is thoroughly proofread to address any such errors.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vhaIobNUXV", "reviewer": "Reviewer_EDLE", "summary": "This work presents a novel approach for visual relationship segmentation that integrates the three critical aspects of a flexible VRS model: standard VRS, promptable querying, and open-vocabulary capabilities. By harnessing the synergistic potential of textual and visual features, the proposed model delivers promising experimental results on existing benchmark datasets.", "questions": "Please refer to Weakness Section for more details.", "limitations": "The authors have thoroughly discussed the limitations of this work.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The authors introduce a flexible framework capable of segmenting both human-centric and generic visual relationships across various datasets. \n2) The authors present a promptable visual relationship learning framework that effectively utilizes diverse textual prompts to ground relationships. \n3) The proposed method shows competitive performance in both standard close-set and open-vocabulary scenarios, showcasing the model’s strong generalization capabilities.", "weaknesses": "1. The experimental comparison suffers from potential unfairness, contrasting the proposed method with others using Focal-L backbone against those employing ResNet backbone seems somewhat unreasonable.\n2. Since this paper includes referring expression comprehension (REC) tasks in abstract, it is unreasonable not to report experimental results on corresponding benchmarks like RefCOCO, RefCOCO+, and RefCOCOg.\n3. While the research perspective of this paper is reasonable and innovative, the overall architecture design of Flex-VRS lacks novelty compared to previous related works.\n4. More visualizations are needed for demonstrating the produced fine-grained masks generated by converting existing bounding box annotations from HOI detection datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful feedback. Please refer to “The General Response to Reviewers” for the reply to the issue of backbone comparison, difference with previous methods and refer to the PDF for the added visualizations of masks generated from bounding box annotations. We respond below to your other comments and questions. \n\n1. **Results on referring expression comprehension (REC) task.**\n    - The referring expression comprehension (REC) tasks on benchmarks like RefCOCO, RefCOCO+, and RefCOCOg are designed to detect objects based on free-form textual phrases, such as \"a ball and a cat\" or \"Two pandas lie on a climber.\" In contrast, the promptable VRS task in our work focuses on detecting subject-object pairs within a structured prompt format, such as <?, sit_on, bench> or <person, ?, horse>, as illustrated in Figure 1 of the main paper. Our model is designed to encode and compute similarity scores for each of these elements separately.\n    - Adapting our model to standard REC tasks would require fundamental changes, as the REC format does not align with the structured, relational queries that our model is designed to handle. Our primary focus is on relational object segmentation based on a single structured query, which differs significantly from the objectives of REC benchmarks. Therefore, it is not reasonable to directly compare our results with those benchmarks without substantial modifications to the model.\n\n2. **The architecture design lacks novelty.**\n\n    - As highlighted in Table 1 of the main paper, our Flex-VRS framework introduces a flexible architecture that supports standard, promptable, and open-vocabulary visual relationship learning. The novelty of our approach lies in the seamless integration of these diverse functionalities within a single, unified system. This design enables flexible and dynamic interactions with visual relationships, setting our work apart from previous methods that often lack such comprehensive capabilities.\n\n    - Furthermore, our model's one-stage design allows it to effectively handle different types of inputs and perform various tasks based on the input, thanks to our novel query mechanism. This innovative query design is a significant departure from previous works, offering a level of adaptability and flexibility that has not been previously developed.\n\n3. **More visualizations of generating masks from bounding box annotations.**\n\n    Thank you for the suggestion. We've included additional visualizations in the attached PDF. These visualizations demonstrate how redundant bounding boxes are consolidated into a single mask, and how our IoU-based filtering effectively removes failure cases.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aLnPjvwdWn", "reviewer": "Reviewer_k2zH", "summary": "This paper proposed a flexible framework that can effectively handle human-object interaction (HOI) detection, scene graph generation (SGG), and referring expression comprehension (REC) tasks. The proposed method further addressed the problem of promptable visual relationship segmentation and enabled the capability for open-vocabulary segmentation. Its main idea is to leverage the pretrained vision-language models for grounding textual features to visual relationship inside images. Experimental results show that the proposed framework is able to achieve SOTA performance on standard, promptable and open-vocabulary visual relationship detection/segmentation tasks.", "questions": "Please refer to the Weakness section.", "limitations": "The authors have widely discussed the limitation of the proposed work and also its challenges in benchmarking.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper is overall well-written and easy to follow.\n\n2. The problem addressed by this paper is novel to me. Unifying various visual relationship detection/segmentation tasks with in a single framework has been seldomly studied in previous works, which might raise new research interests in this field.\n\n3. The overall performance of the method is competitive. It is able to achieve SOTA results on most VRD tasks and outperform previous methods notably.", "weaknesses": "1. According to Table 7, the major performance improvement of the proposed method comes from the supervision of the mask head, which is provided from SAM and never used by previous works. When only the box head is used, the results of the proposed method is behind the current SOTA methods notably on HICO-DET and VCOCO. This makes the benefit of the proposed unified framework questionable if such large performance drops are observed.\n\n2. Apart from introducing the segmentation head, the major technical contribution of the proposed method is unclear to me. The model architecture and losses are very similar to those previously proposed dual-decoder architectures in GEN-VLKT, RLIP, etc., and this part needs further clarification.\n\n3. In Table 7, from the last row, we can see that the benefit brought by adding PSG is minor. Hence, the results on PSG dataset should also be reported so that the readers could have a better understanding on how unifying the framework could benefit each individual task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and helpful feedback. Please refer to “The General Response to Reviewers” for the reply to the issue of the difference with previous methods and refer to the PDF for the added tables for PSG results and mask head only results. We respond below to your other comments and questions. \n\n1. **Major performance improvement comes from the mask head.**\n\n    This performance drop can be attributed to the fact that our network architecture, derived from Mask2Former, is optimized for pixel-wise segmentation tasks and does not fully leverage the benefits of box-based supervision.\n    To further investigate, we trained existing HOI detectors (CDN, STIP, GEN-VLKT) with only mask head, using segmentation masks instead of box annotations. Surprisingly, all of these approaches led to reduced accuracy on HICO-DET as in Table 2 of the attached PDF, indicating that their networks’ performance with segmentation masks are not as effective as with box annotations. We argue that our proposed method’s preference for mask annotations is analogous to how previous methods favor box annotations. \n\n2. **The technical contribution of the proposed method is unclear.**\n\n    While our architecture shares similarities with dual-decoder models like GEN-VLKT and RLIP, our key contributions are: \n\n    (1) ***Unified Framework***: As in Table 1 of the main paper, our model integrates standard, promptable, and open-vocabulary VRS tasks into a single system, offering broader flexibility than previous methods. \n\n    (2) ***Mask-Based Approach***: We utilize masks to handle various VRS tasks, allowing our model to adapt to different types of annotations effectively, including HOI detection and generic scene graph generation.\n\n    (3) ***Dynamic Prompt Handling***: Our approach supports dynamic prompt-based and open-vocabulary settings, addressing limitations of fixed-vocabulary models. As in the Figure 4 of the main paper, our model can even combine the promptable query with open-vocabulary ability to make the model ground novel relational objects.\n\n     Please refer to “The General Response to Reviewers”, where we elaborate more on the difference of our model with previous works.\n\n3. **Results of PSG in Table 7 of the main paper.**\n\n    Thank you for your suggestion. We have added the results for the PSG dataset in Table 7 of the main paper, please refer to the PDF. The updated results show that while adding HICO-DET and VCOCO can provide slight improvements for PSG, PSG itself does not significantly enhance performance on HICO-DET. This is due to the relatively small size and distinct distributions of these datasets, which may not always be mutually beneficial. We appreciate your input and hope this additional information clarifies the impact of unifying the framework across different tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful feedback. Our proposed Flex-VRS is commended for its ability to support a variety of tasks(Reviewer LLLd, EDLE, k2zH), its integration of flexibility and open-vocabulary capability into the VRS model(Reviewer LLLd, EDLE), its competitive performance(Reviewer EDLE, k2zH), and its clear presentation(Reviewer LLLd, k2zH). We address some common questions from reviewers here and will incorporate the feedback in the revision. Please also refer to the attached PDF for added visualizations of generated masks and tables for PSG results and mask head results.

1. **Difference from previous methods (e.g. UniVRD).**

    We would like to clarify the contributions and differences between our method and previous work, including UniVRD and RLIPv2.

    - **Comparison with UniVRD** – *Methodology Difference*: UniVRD uses a two-stage approach, where the model first detects independent objects and then decodes relationships between them, retrieving boxes from the initial detection stage. In contrast, our method employs a one-stage approach, where each query directly corresponds to a <subject, object, predicate> triplet. This transition improves time efficiency from O(MxN) to O(K), where M is the number of subject boxes, N is the number of object boxes, and K is the number of interactive pairs. ***Our approach also provides greater flexibility by learning a unified representation that encompasses object detection, subject-object association, and relationship classification in a single model.*** (As shown in Figure 1, the subject-object pairs need to be accurately localized and understood.) Our model is superior in terms of model capacity and data efficiency. While we use much fewer training data (x50 less, without using VG and Objects365) and our model with the Focal-L backbone is much smaller than UniVRD (164M vs 640M) with LiT(ViT-H/14), we achieve comparable results(37.4 vs 38.1 on HICO-DET).

     - **Performance Comparison with RLIPv2** – *Scaling and Design*: While our method does not match RLIPv2 in performance, this is due to different design philosophies and goals. RLIPv2 is a two-stage approach optimized for large-scale pretraining and relies on separately trained detectors. ***Our model, however, is not designed for pretraining and does not include a separately trained detector. Our focus is on enhancing the flexibility of the VRS model without relying on extensive curated data(x50 more, VG and Objects365).*** Thus, the differences in performance are attributed to the scale and design objectives rather than a direct comparison of methods.

2. **FLOPs and the number parameters of the backbone compared to ResNet.**

    As in Table 2 and 3 of the main paper, we have done extensive comparisons with previous methods, including backbones on ResNet-50/101/152, EfficientNet, Hourglass, Swin Transformers, and LiT architectures. For previous methods that utilize ResNet backbone for HOI detection and PSG, we already conducted a thorough comparison and included (VSGNet, ACP, No-Frills), which used ResNet-152.  To the best of our knowledge,  larger ResNet, such as ResNet-200, and ResNet-269, are not used in previous methods on related tasks. ResNet, we have included the largest model ResNet-152, which has 65M parameters and 15 GFLOPs.

    Other baselines are not using the ResNet backbone, for example, UniVRD is using LiT(ViT-H/14) backbone. It has 632M parameters and 162 GFLOPs, a lot more than our198M parameters and 15.6 GFLOPs,  but still performs worse than our model.

Meta-review: This paper was reviewed by four experts in the field. The authors did a great job in the rebuttal, and reviewers raised their ratings after the rebuttal (unanimously agreed to accept the paper). The reviewers did raise some valuable suggestions in the discussion that should be incorporated in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.

Why not higher: Reviewer k2zH raised some valuable concerns, including more discussions on the unified framework for VRS tasks and using SAM-based models to compare with other models trained on much smaller datasets.
Decision: Accept (poster)