Scene Graph Generation with Role-Playing Large Language Models
Authors: Guikun Chen
Jin Li
Wenguan Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: Scene Graph Generation
Large Language Model
Primary area: machine_vision

Abstract: Current approaches for open-vocabulary scene graph generation (OVSGG) use vision-language models such as CLIP and follow a standard zero-shot pipeline – computing similarity between the query image and the text embeddings for each category (i.e., text classifiers). In this work, we argue that the text classifiers adopted by existing OVSGG methods, i.e., category-/part-level prompts, are scene-agnostic as they remain unchanged across contexts. Using such fixed text classifiers not only struggles to model visual relations with high variance, but also falls short in adapting to distinct contexts. To plug these intrinsic shortcomings, we devise SDSGG, a scene-specific description based OVSGG framework where the weights of text classifiers are adaptively adjusted according to the visual content. In particular, to generate comprehensive and diverse descriptions oriented to the scene, an LLM is asked to play different roles (e.g., biologist and engineer) to analyze and discuss the descriptive features of a given scene from different views. Unlike previous efforts simply treating the generated descriptions as mutually equivalent text classifiers, SDSGG is equipped with an advanced renormalization mechanism to adjust the influence of each text classifier based on its relevance to the presented scene (this is what the term “specific” means). Furthermore, to capture the complicated interplay between subjects and objects, we propose a new lightweight module called mutual visual adapter. It refines CLIP’s ability to recognize relations by learning an interaction-aware semantic space. Extensive experiments on prevalent benchmarks show that SDSGG significantly outperforms top-leading methods.

Scene Graph Generation with Role-Playing Large Language Models
Guikun Chen
Jin Li
Wenguan Wang
ZZejiaang University
National Key Laboratory of Machnne Hybrid Augmented Intelligence, Xi'an an Jiaotong University
Changsha University of Science & Technology
https://gi com/ hte//gukkun guikunchen/SDD00
Abstract
Current approaches for open-vocabulary scene graph generation (OVSGG) use laiiun- models such as CLIP and follow a standard zero-shot pipeline- computing similarity between the query image and the text embeddings for each category (i.e., text classifiers). In this work, we argue that the text classifiers adopted by existing OVSGG methods, i.e., category-/part-lew prompts, are scene- agnostic as they remain unchanged across contexts. Using such fixed text classifiers not only struggles to model visual relations with high variance, but also falls short in adapting to distinet contexts. To plug these intrinsic shortcomings, we devise SDSGG, scene-specific description based OVSGG framework where the weights text classifiers are adaptively adjusted according to the visual content. In particular, to generate comprehensive and diverse descriptions oriented to the scene, an LLM asked to play different roles (e.g., biologist and engineer) to analyze and discuss the descriptive features of a given scene from different views. Unlike previous efforts simply treating the generated descriptions as mutually equivalent text classifiers, SDSGG is equipped with an advanced renormalization mechanism to adjust the influence of each text classifier based on its relevance the presented scene (this is what the term "specific" means). Furthermore, capture the complicated interplay between subjects and objects, we propose a new lightweight module called mutual visual adapter. It refines CLIP's ability recognize relations by learning an interaction-awant semantic space. Extensive experiments on prevalent benchmarks show that SDSGG outperforms top-leading methods by clear margin.
Introduction
SGG [1] aims create a structured representation of image by identifying objects as nodes and their relations as edges within a graph. The emerging field of OVSGG [2,3], which broadens the scope of SGG to identify and associate objects beyond predefined set of categories, has become research hotspot for its prospective to amplify the practicality of SGG in diverse real-world applications.
OVSGG has achieved notable progress due to the success of vision-language models (VLMs) [4, 5] and prompt learning [6, 7]. Existing OVSGG methods adopt standard zero-shot pipeline [4], which computes similarity between the visual embedding from query image and the text embedding from pre-defined text classifiers (c.f. Fig. la). One straightforward direction for OVSGG is to use only the category name (e.g.. "riding") [2, 8, 9, 3, 10] the text classifier and perform vision-land -angunla
'The first two authors contribute equally to this work. Corresponding Author: Wenguan Wang.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a)
visual embedding
text embedding
CLIP
eating riding
Image Encoder
compute similarity choose highest
Text Encoder
parked
(b)
What are the useful visual features of the subject and object that can distinguish the relation "riding""
Persona Identification
Participants: Experts specialized in biology physics
and engineering
with legs
Task Setup
subject
with bands
Q: Imagine there is human that is riding, ask yourselves to have detailed discussion about the comprehensive descriptions the scene.
with four legs with a saddle
object
Multi-persona persona Collaboration
From biological standpoint, we can infer that the sub- ject might be holding onto the object's back or using some form of hamess for stability during the ride.
(c)
imagining
equipment like rain may be used
the physics involved, there would likely be motion involved, with the subject and object moving together.
subject holding onto the object
motion would be involved
engineering perspective, any mechanical interac- tions between een the human and the object, like reins.
Figure 1: Illustration of the used text classifiers in OVSGG. (a) CLIP performs zero-shot classification by computing similarity between the query image and the text embeddings for each category, then choosing the highest. (b) To further utilize the learned semantic space CLIP, one can compute similarities multiple part-level prompts (e.g.. the object of (man, riding, horse) may be described with "with four legs" and "with a saddle"). (c) Instead of using these scene-agnostic text classifiers. SDSGG adopts comprehensive, scene-specific descriptions generated by LLMs, which can adapt to specific contexts by using the proposed renormalization.
alignment in prompt learning to learn the underlying patters.. On the other hand, [11, 12] argue that such methods fail to utilize the rich context of additional information that language can provide. To address this, [12] decomposes relation detection into several separate components, computing similarities by checking how well the visual features of the object match the part-level descriptions As shown in Fig. Ib, the object of relation "riding" should have four legs, saddle, etc.
Despite these technological advances, we still observe current OVSGG systems lack in-depth in- spection of the expressive range of the used text classifiers, which puts performance cap them. Concretely, OVSGG models that rely on scene" aggossicc text classifiers have the following flaws. First, methods based category names [2, 3] struggle to model the large variance in visual relations. Using only category names as classifiers [4] does hold water when applied to object recognition [13]. For instance, CLIP shares common visual features across diverse image-text pairs of tigers which encompasses a variety of tiger appearances and corresponding descriptions. Nonetheless, the scenario becomes far more complex when it comes to relation detection. The visual features that define the rela- tion "on" can vary dramatically across scenes, e.g., "dog on bed" vs. "people on road". RECODE [12] proposes to decompose relation detection into recognizing part-level descriptions for both subject and object, hence partially easing the aforementioned difficulty. Yet, it computes similarities for the subject and object separately and does not model the interplay between subjects and objects. Second, pert-level prompt based methods uniformly process all descriptions as affirmative classifiers [11, 12], overlooking the possibility that some text classifiers might be contrary to specific contexts. When querying LLMs for distinctive visual features of subjects and objects to distinguish the predicate "riding". with the subject as "human" and the object as "horse". LLMs provide part-level descriptions of the expected appearance of both entities. All generated descriptions are treated equally as definitive text classifiers. However, these descriptions could potentially be misleading, as LLMs produce them without considering the specific context, even resulting in some descriptions that are wholly irrelevant to the presented image. For example, LLMs typically associate the predicate "riding" with the animal
I 'The terms "description" " and "prompt" are used interchangeably, all denoting text. classifiers.
A scene typically involves objects and their interplay and relationships.

---

"with four legs". Nonetheless, such associations are indeed irrelevant, as an animal's legs are not always visible in the presented image, or the animal may have only two legs.
Filling the gaps identified above calls for a fundamental paradigm shift: moving from the fixed, scene-agnostic (ie., category-/part-levell text classifiers towards flexible, scene-specific ones. In light of this, we develop SDSGG, a scene-specific description based OVSGG framework that utilizes text classifiers generated from LLMs, complemented by a renormalization technique, to understand scenes from different perspectives, For the textual part, given a scene with specified content, an LLM assigned distinct roles, akin to experts specializing in biology, physics, and engineering. to analyze descriptive scene features comprehensively (cff. Fig. Ic). Such a multi-person persona scheme designed to improve the diversity of the generated scene descriptions as LLMs tend to generate repetitive content. LLM can be queried multiple times to obtain large number of scene descriptions. Moreover, since not all descriptions are relevant to the presented image (e.g., some parts of the object may not appear), SDSGG is equipped with an advanced mechanism that renormalizes each scene description via opposite descriptions corresponding to the original descriptions. This involves evaluating two vision- vision-language similarities: one for the original scene description and the other for its opposite. The difference between the two similarities is viewed as the self-normalized similarity of the scene description, allowing for flexible control over its influence. For instance, an irrelevant description would yield a self-normalized similarity close to zero, as the two similarity scores of it and its opposite would be very close. By doing so, the generated scene-level descriptions become flexible, scene-specte speciffc descriptions (SSDs). For the visual part. we propose new adapter for relation detection, called mutual visual adapter, which consists of several lightweight learnable modules. The proposed adapter projects CLIP's semantic embeddings into another interaction-awarc space, modeling the complicated interplay between the subject and object through cross-attention.
With the proposed adaptive SSDs, our SDSGG is capable of: i) adapting to the given context via evaluating the self-normalized similarity of each SSD; ii) alleviating the overfitting problem in OVSGG models [2, 3] that use only one classifier; and iii) naturally generalizing to novel relations by associating them with SSDs. We validate SDSGG on two widely-used benchmarks, i.e. Visual Genome (VG) [14] and GQA [15]. Experimental results show that SDSGG outperforms existing OVSGG methods [3] by a large margin. The strong generalization and promising performance of SDSGG evidence the great potential of scene-specific description based relation detection.
2
Related Work
Scene Graph Generation. Since [1] introduces iterative message passing for SGG, research studies in structured visual scene understanding have witnessed phenomenal growth. Tremendous progress has been achieved and can be categorized into: i) Two-stage SGG [16, 17, 18, 19, 20]. which first detects all objects the images and then recognizes the pairwise relations between them; ii) Debiased SGG [21, 22, 23, 25, 26, 27, 28], which focuses the problem of long-tailed predicate distribution in the current dataset; fii) Weakly-supervisc. SGG [29, 30, 31, 32, 33], which investigates how to generate scene graph with only image-level supervision; iv) One-stage SGG [34, 35, 36, 37, 38, 39], which implements SGG within an end-to-end framework (also exemplified in other relation detection tasks [40, 41, 42]), discarding several hand-crafted procedures; v) Open-vocabular SGG, which learns to recognize unseen categories during training by using category-level [2, 8, 9, 3, 10] or part- level prompts [12]. vi) Few-show SGG, which learns to recognize relations given few examples [43].
Existing OVSGG frameworks adopt a standard open-vocabulary learning paradigm, i.e., perform vision-language alignment in the pre-trained or random initialized semantic space with supervision of only the category names. One except [12] reformulates OVSGG from recognizing category- prompts into recognizing pert- prompts, by decomposing SGG into several separate components and computing their similarities independently, in a training-free manner. SDSGG represents the best of both worlds. On the one hand, we point out the drawbacks of the commonly used scene-agnostic text classifiers and introduce scene-specific alternates to understand scenes from different perspectives. On the other hand, SDSGG incorporates learnable mutual visual adapter capture the underlying patterns in the dataset and proposes to renormalize text classifiers for adapting to specific contexts.
Learning. Most deep neural networks operate on the close-set assumption, which can only identify pre-defined categories that are present the training set. Early zero-shot learning approaches [44, 45, 46] adopt word embedding projection to constitute the classifiers for
3

---

unseen class classification. With the rapid progress of vision language pre- pre-training [4, 5, 47], open vocabulary learning [48] has been proposed and demonstrates impressive capabiliitess by, for example, distilling knowledge from VLMS [49, 50, 51, 52, 53, 54], exploiting caption data [55, 56], generating pseudo labels [57, 58, 59, 60, 61], training without dense labels [62, 63, 64), joint learning of several tasks [65, 66, 67], and training with more balanced data [68, 69].
While sharing a very high-level idea of vision-language alignment in open-vocabulary methods, our SDSGG i) explicitly models the context-dependent scenarios and introduces scene-specific text classifiers as the flexible learning targets, and ii) incoroperates new mechanism for computing self-normalized similarities, thereby renormalizing text classifiers according to the presented image.
VLMs Meet LLMs [70]. The big win for VLMs has been all about getting the model to match up pictures and their descriptions closely while keeping the mismatched ones apart [4, 5, 47]. This trick, inspired by contrastive learning from self-supervised learning [71, 72, 73, 74, 75], helps VLMs get really good at figuring out what text goes with what image. Moreover, prompt learning acts as a flexible way to communicate with VLMs. giving them nudge or context to apply their knowledge of images and text in specific ways [6, 76, 77, 7, 78]. In addition to hand-crafted or learnable prompts, [11] offers fresh perspective, i.e., using LLMs to generate detailed, comprehensive prompts as the inputs of VLMs' text encoder. Many follow-up works [79, 80, 81, 82, 83, 84, 85] across various domains and tasks demonstrate the effectiveness of integrating VLMs and LLMs.
Category-/part-leve prompts are scene-agnostic and cannot adapt to specific contexts. To this end, SDSGG adopts scene-specific descriptions, generated by LLMS in multi-persona collaboration fashion, as the inputs of CLIP's text encoder. Different from part-level prompt based approaches [1] 12] which processes all part-level prompts affirmative classifiers, SDSGG provides a flexible alternative via the association between classifiers (i.e., SSDs) and categories, and the renormalizing strategy w.E. each SSD. Since the learned semantic space of VLMs may not be sensitive to relations [12], we design a lightweight mutual visual adapter to project them into interaction-aware space for capturing the complicated interplay between the subject and object.
Methodology
Task Setup and Notations. Given an image I, SGG transforms it into a structured representation, i.e. a directed graph G = {O,R) R2, where O represents localized (i.e., bounding box) objects with object category information and represents pairwise relations between objects. For a fair comparison, this work focuses on predicting R given O i.e., the predicate classification task which avoids the noise from object detection, as suggested by [1, 3, 12]. Our study delves into the intricacies of transitioning SGG from traditional closed-set set setting to an open vocabulary paradigm. This transition enables the system to recognize previously unseen predicate categories (i.e., novel split) by lcarning from observed predicate categories (i.e., base split) during training.
SDSGG follows the standard zero-shot pipeline with VLMs [4], which computes similarity between the visual embedding e and the text embedding & for each category, and the category with highest similarity is viewed as the final classification result (c.f. Fig. 2a). For each subject-object pair, v can be derived by feeding cropped patches from the input image I into the visual encoder. The text embedding used existing OVSGG frameworks falls into two main settings: i) Each category consists of only one text classifier, i.e., the category name itself. ii) Each category consists of multiple text classifiers w.r.t subject and object, i.e., pert-level descriptions. SDSGG reformulates the text classifiers as scene-specific descriptions which will be detailed in $3.1.
Algorithmic Overview. SDSGG is SSD based framework for OVSGG, supported by the coopera- tion of VLMs and LLMs. For the textual part (c.f. Fig. 2b), SDSSG enjoys the expressive range of the comprehensive SSDs generated by LLMs' multi-persona collaboration. This is complemented by renommalizing mechanism to adjust the influence of each text classifier. For the visual part (c.f. Fig. 2c), SDSSG is equipped with mutual visual adapter to aggregate visual features v from for a given subject-object pair. After introducing how we generate and use SSDs for the text part ($3.1) 1) and the mutual visual adapter for interplay modeling of the subject and object ($3.2), we will claborate on SDSGG's 's training objective ($3.3).

---

(a)
Mutual Visual
laying on riding
Similarity
Adapter
looking at
(b)
(c)
Directional Marker
Self-normalized Similarity Measurement: (Eq. 2)
(Eq. 6)
d' two more objects partially overlap
Muhual Visual Adapter (MVA)
Scera Scere specife Descaiption
d each obj oct is completely separate
Certaun coexistence
fo
de on a road
K Q
kmg
possible Eoexistence
di in non- road environment
c development of leeves, stem, etc.
contradiction
lacks any botanical features
Visual
Encoder (Eq. 3)
Linear Layer
attention
cross
add & pooling
concale
Figure 2: (a) Overview of SDSGG. (b) Each text classifier of SDSGG contains raw description d" and an opposite description d'. As such, the self-normalized similarities can be computed with the association (CP) between predicate categories and SSDs. (c) Given the visual features (i.e., fime, foda and fa of both subject and object extracted from CLIP's visual encoder, our mutual visual adapter (MVA) projects them into interaction-awes aware space and models their complicated interplay with cross- attentinen
3.1 Scene-specific Text Classifiers
Scene-level Description Generation via Multi- Collaboration. Using standard prompts to query LLMs a direct way to generate scene descriptions. For instance, one could straightforwardly prompt LLM with question like "Imagine there is an animal that is eating, what should the scene look like?". L.M's response would typically sketch out an envisioned scene based on its statistical training large corpus. However, these responses may not fully capture the scene's 's complexity, often overlooking aspects such as the spatial arrangement of elements and the background environment.
To alleviate this, we draw inspiration from recent advances in LLMs' multi-persona capabili- ties [86, 87, 88, 89]. Specifically, LLM adopts three distinct roles, mirroring the expertise found in experts specializing in biology, physics, and engineering. This approach allows for compre- hensive discussion of what given scene entails. Because each query LLM usually only yields 3-5 sentences of description, we query LLM several times, each time giving LLM a different scene content to be discussed, thus obtaining a large number of scene descriptions. Since these initial de- scriptions may suffer from noise and semantic overlap, we ask LLM to streamline and combine these descriptions, ensuring more cohesive and distinct scene-level descriptions D; (d) dd ... .d" and corresponding text embeddings T tt,, ......,, t.v J. where N denotes the number of SSDs and text embeddings T are extracted by the text encoder of CLIP. Due to the limited space, we provide more details and prompts for generating scene descriptions in the appendix (SD).
Association between Scene-level Descriptions and Relation Categories. So far, we have obtained various scene-level descriptions that have the ability to represent diverse scenes. A critical inquiry arises regarding their utility for relation detection, given their lack of explicit association with specific relation categories. To address this, we delineate three distinct scenarios characterizing the interplay between relation categories and scene descriptions: i) certain coexistence (C.- 1), where a direct correlation exists; ii) possible coexistence (C = 0), indicating a potential but not guaranteed association; and iii) contradiction (C.- = 1)), denoting an incompatibility between the scene description and relation category. Here Chh denotes the correlation between relation E R. and neh scene description, and generated by LLMS (prompts are shown in $D). Such categorization enables to calculate the similarity for each relation category:
sim(v,r) =>' C. (v, tt).
(1)
where (.-)) denotes the cosine similarity with temperature [4].
5

---

Scene-specific Descriptions Scene-level Descriptions + Reweighing. Upon examining text classifiers depth, we noticed that certain classifiers are contextually bound, egg., "two or more objects partially overlap each other" may not exist in all scenes. This observation underscores the necessity for a mechanism to evaluate the significance of each text classifier, rather than applying uniform weight across the board. This is exactly what the term "specific" means. Recall that the similarity measurement between an image and the text "a photo of a cat" alone yields limited insight. However, when juxtaposed with multiple texts, such as "a photo of a cat/dog/tiger", the comparison of similarity scores across these categories reveals which category (cat, dog, or tiger) the image most closely resembles. Inspired by this, we propose the incorporation of an opposite description dă ($D) for each raw scene description dy as reference point (e.g., "two or more objects partially overlap each other" vs. "each object is completely separate with clear space between them"), resulting in SSDs D.- {(dh, dy),
()) (de. de),... (dd" . }} and updated text embeddings T (1h 12), (ta (2)............. ... (42 1/ }) Subsequently, the self-normalized similarity is defined as:
sim(u, r) - C² ((
(2)
The difference in similarity scores, i.e., (v, (2 (w, ty), quantifies the relative contribution of that SSD. By such means, SSD irrelevant to the presented context will have minimal effect, as the similarity scores of it ((v, ) and its opposite ((v, ( would be nearly identical.
3.2 Mutual Visual Adapter
After introducing how to obtain the text embeddings and how to compute vision-language similarity, one question remains at this point: how to obtain visual embeddings? When given a subject-object pair with bounding boxes from O there exist various strategies for aggregating visual features for the subject and object within For example, traditional closed-set SGG frameworks [16, 17] employ Rol pooling extract visual features for specified bounding boxes, subsequently fusing these features for further classification. In contrast, the more recent OVSGG framework [12] uses the visual encoder of CLIP to extract visual embeddings of both subject and object. Then, it processes two visual embeddings independently through part-level descriptions. Such an independent approach, however, overlooks the informative interplay between the subject and object.
To address this oversight and capture the complicated interactions between subject and object, we introduce new component: the mutual visual adapter (MVA). MVA is composed of several lightweight, learnable modules designed to fine-tune CLIP's visual encoder specifically for pairwise relation detection. This approach aims to enhance the model's s ability to recognize the nuanced interactions that define relationships between subject and object in an image.
Regional Encoder. Given an image I and subject-object pair with bounding boxes (b,, and ha) from O, the initial visual embeddings can be obtained from the visual encoder of CLIP:
fajo ff 1770 1f%/ fe
M f Encoder, (Crop(I, bejo));
(3)
where M denotes the number of patches, Encoder, is CLIP's visual encoder that is kept frozen during training, and Crop represents image cropping.
Visual Aggregator. Next, MVA is adopted to aggregate f. and fo by cross-attention and two lightweight projection modules. Let the subject part be the query, and the object part be the key and value. The patch embeddings of object fimmg are first projected into low-dimensional, semantic space:
(omg Linear (Je"").
(4)
where Linear denotes a standard fully connected layer. Afterwards, cross-attention is adopted to capture the complicated interplay between subject and object, resulting in an aggregated visual embedding for the given subject-object pair:
V so Linear (AvgPool(LM + (f (
(5)
where AvgPool is the average pooling. LN is the standard layer normarlization. CrossAttn( Q, KV) denotes the standard cross-attention operation. tos can be computed in similar way by exchanging the query and key of cross-attention. Combining them together leads to the final visual embedding v (v.0 + //x for final similarity measurement. As such, MVA captures the interplay of subject and object in the projected, interaction- space.

---

Directional Marker. One may notice that the structure of MVA is symmetric and has no information about which imput branch is the subject/object. This has a relatively small effect on semantic relations, but significant effect on geometric relations. For instance, after exchanging the location of the subject and object (image flipping), the relation "eating" remains unchanged, while the relation "on the left" would become "on the right". Here we simply incorporate two text embeddings (t, and t.) of "a photo of subject/object" into MVA and thus update the visual embedding as:
Uso = Linear (Concate(Vse; tyl);
(6)
where Concate denotes concatenation. Vos and can be updated accordingly. Further exploration of directional marker, e.g., incorporating more complex feature fusion modules, is left for future work.
3.3 Training Objective
typical training objective open-vocabulary learning aims to bring representations of positive pairs closer and push negative pairs apart in the embedding space. In SDSGG, the term "positive/negative pairs" is not defined at the category level but at the description level, requiring losses tailored for different relation-description association types. Given a labeled relation triplet, one simplest contrastive loss can be defined as:
L = T E EcTET
(0, (v,
a+C
(7)
similarsty
target
where a is a scaling factor. However, as for scene descriptions marked by possible coexistence (Le., C = 0). there is no direct target that can be used for training. Inspired by the identical mapping in residual learning [90], we make the prediction results of MVA close to those of CLIP. As such, MVA can learn the implicit knowledge embedded CLIP's semantic space. In addition, this regularization term prevents MVA from overfitting to relations in the base split, which is a common problem in open-vocabulary learning. Hence, the loss is further reformulated as:
L ELET (0,t)))(0r (0,
C. (3=simcLip(4,re) rel) A)
(
similarity
target
where / is another scaling factor, siMCLIP rel) denotes the Disiun-language similarity derived from the original CLIP, and λ is a constant scalar and is empirically set to 3c-2.
Experiment
4.1 Experimental Setup
Dataset. We evaluate our method on GQA [15] and VG [14] following [3, 12].
Split. Following previous work [3]. VG is divided into two splits: base and novel split. The base split comprises 70% of the relation categories for training, while the novel split contains the remaining 30% categories invisible during training. For a more comprehensive comparison, we also conduct testing on the semantic set, encompassing 24 predicate categories [16, 12] with richer semantics. base and novel split of GQA [15] are obtained in similar manner ($B).
Evaluation Metrics. We report Recall @K (R @ K) and Recall @ K(mR @ K) following [3, 22].
Base Models and Competitors. As for the base and novel split, we compare SDSGG with two baselines: i) CLS [4], which uses only the category-level prompts to compute the similarity between the image and text; and ii) Epic [3], a latest OVSGG method, which introduces an entangled cross- modal prompt approach and learns the cross-modal embeddings using contrastive learning. In terms of the semantic split, we compare our SDSGG with three baselines: i) CLS [4], which uses only the cetegory-level prompts: ii) CLSDE [12], which uses prompts of relation class description; and iii) RECODE [12], which uses visual cues of several separate components. Since [10] has neither released the detailed split nor the code, it is not included in the comparisons for fairness.
Implementation Details. Due to limited space, implementation details are left in the appendix ($B).
4.2 Quantitative Comparison Result
We conduct quantitative experiments on VG [14] and GQA [15]. ensure the performance gain reliable, each experiment is repeated three times. The average and standard deviation are reported.
7

---

Table 1: Quantitative results ($4.2)
VG [14]
base and novel.
Method
Split
20
R@
R@ TOOT
mR@20T
mR@
mR 100T
CLS [
2.1
3.2
3.9
7.0
9.0
10.9
Epicneevas [3]
base
22.6
27.2
Ours
18.7
20.5
31.6
9.2
12.4
14.8
CIS [
13.2
18.1
22.2
11.5
17.9
23.8
Epicneevar
novel
7.4
9.7
Ours
18.4 -0.53
25.4:00.88
29.6
+0.44
25.2.00.5
31.2
Table 2: Quantitative results (84.2) on VG [14] semantic.
Method
@@2
R 501
R@ 1001
mR @20
mR 501
mR@ 1001
CLS CLS
[4]
7.2
10.9
13.2
9.4
14.0
17.6
[LSDE/Nuutess 12]
7.0
10.6
12.9
8.5
13.6
16.9
RECODE [NouiPS22
12]
7.3
11.2
15.4
8.2
13.5
18.3
RECODE (Neuttss)
12]
9.7
14.9
19.3
10.2
16.4
22.7
RECODE [NaulPS23 [12]
10.6
18.3
25.0
10.7
18.7
27.8
Ours
20.4
29.3
34,9
40.06
16.8
10.08
22.7
-0.41
28.4
VG [14] base and novel. Table illustrates our compelling results over the latest OVSGG model, Epic [3]. Since Epic did not release the full code, we report values under the same setting as in the original paper. CLS [4] leverages the original CLIP's checkpoint (w/o fine-tuning) and only the category-level prompts. The performance of CLS on the novel split is better than that on the base split because i) the base split contains more relation candidates be classified (35 vs. 15); and ii) the base split contains head, uninformative relations (e.g., "on" and "has") which are hard to be distinguished by CLIP [12]. By perform contrastive learning on the base split to learn the entangled cross-modal prompt, Epic [3] achieves significant performance gains the base split (i.e., 22.6% vs. 3.2% R@50 and 27.2% vs. 3.9% R 100). However, Epic demonstrates worse performance on the 18 R 50 and 9.7% 22.2% @ 100), which indicates its overfitting
novel split (i.e.. 7.4% on the training data. By incorporating SSDs, SDSGG outperforms Epic by large margin on both the base split and novel split. For instance, SDSGG exceeds Epic by 3.9%/4.4% R @ 50/100 on the base split and 18.0% 50/100 on the novel split, respectively. The significant performance improvements the novel set demonstrate the strong zero-shot capability of our approach. In addition, our margins over the CLS are 16.6% R@K K and 2.2% mR @ K on the base split, and 5.2% R@K K and 5.6%~7..4% mR @K the novel split, respectively.
VG [14] semantic. In Table 2, we present the numerical results of SDSGG against the latest OVSGG work [12] on the semantic split. By leveraging pert-level prompts, RECODE demonstrates superior performance compared to CLS [4] and CLSDE [12]. The introduction of filtering strategies, i.e., RECODE", shows substantial improvements. As seen, SDSGG surpasses all counterparts with remarkable gains on all metrics. In particular, SDSGG exceeds RECODE by 10.9% mR @ 20/50/100. Notably, SDSGG achieves impressive
on @@ 20/50/100, and 6.1 111%/400%/066.. performance without relying on additional augmentation or data [10]. Since an increase on mR implies the average of the improvements for all categories, yields mR @K may be relatively lower than that on R@K. K. Without bells and whistles, SDSGG establishes a new state-of-the-ar..
GQA [15] base and novel. Since the codes of Epic [3] and RECODE [12] are insufficient to support replication, only compare SDSGG with CLS on GQA [15]. As shown in Table 3, SDSGG outperforms CLS by a large margin across all splits and metrics. Since CLS is not trained on the base split, its performance on R@K K is rel- atively low due to the massive annotations of geometric relations that not semantically rich [12].
Table Quantitative results GQA [14] base & novel.
Method
Split
R@20/500100
@ m2220/50/100
[LS
4.2/6.47.9
8.9/13.2/15 15.3
base
33.4/43.9 /4,55
21.0/ 24,5 /
Ours
21.10/21.21/11100
20.17/20.566 20.79
CLS [4]
21.3/ / 32.1
16.6 27.0/ 29,4
27.2/37.4
23.8 / 32.8 37.3
Ours
#0.20/#1.55/44 41.4
Taking together, our extensive results provide solid evidence that SDSGG successfully unlocks the power of LLMs in OVSGG, and SSD is promising alternative to category nategorr-/part-levell prompts.

---

CLIP
rkling
42.4%
sanding
16.7%
hanging from
walking on
18.8%
mounted on
1945
atlached to
Sping
usig
SDSG0
SDSGG
hanging from
$5.4%
lollnn
35.2%
mounted on
18.45
plasigg
23.05
stire, hinging from, polc>
lying on sttached
Sman, holding, nacket>
carrying
weng
Figure 3: Visual results ($4.3) on VG [14].
4.3 Qualitative Comparison Result
Fig. visualizes qualitative comparisons SDSGG against CLIP [4] on VG [14]. As seen, with the proposed SSDs, SDSGG generate higher-quality relation predictions in challenging scenarios. We respectfully refer the reviewer to the appendix ($E) for more qualitative comparisons.
4.4 Diagnostic Experiment
For thorough evaluation, we conduct series of ablative studies VG [14].
Textual Part. We first study the effec- tiveness of our multi persona collaboration
Table
Ablation studies ($4.4) on MPC.
Method
Split
R#20/501100
mR 20/50/100
(MPC) for scene-specific description gen-
Ours
18.7 31.6
12.312.//
eration ($3.1) in Table 4. Here we use the standard prompts to query LLMs for generating descriptions. As seen, without MPC, the performance drops drastically, e,g., 11.8%/1 1.8% vs. 31.6%/30.00 .0% R@ 100 on the base/novel split, respectively. This indicates importance of the text classifiers used as they have impact on both training and testing.
who MPC
base
6.0/9.3/11 11.8
5.5/
Ours
18.9725.87 30.0
novel
w/o MPC
5.2/8.7 11.8
4.9/ 11.0/ 16.2
While the effectiveness of MPC has been validated, one may wonder: i) Why use these three roles? ii) How to ensure the completeness and quality of generated classifiers? We want highlight that: Different roles are used to increase the variety of descriptions. There is no word on exactly which roles should be used. ii) These open problems are beyond the scope of this work. We leave them for future work. iii) This work makes the first attempt to enhance classifier generation for OVSGG via MPC. iv) The experimental results suggest that the current generated SSDs are good enough for commonly used relation categories. To provide more empirical results, we investigate the impact of proposed self-normalized similarities and the number of used SSDs in the appendix ($A).
Visual Part. Then, we examine the impact of mutual visual adapter (MVA, $3.2) and directional marker (DM, $3.2) in Table 5. The 1 row denotes strong baseline, i.e., a multi-layer perceptron with comparable parameters to aggregate the visual features. Upon projecting the visual features into interaction-aware space and applying cross-
Table 5: Ablation studies ($4.4) on the visual part.
MVA
DM
Split
R 20/50/100
mR 20/50/100
13.1/ 22,44
8.6/116 13.8
base
17.0/ 28..2
9.2/123 14.6
/1.6
9.1/12.3 14.7
17.4/24.7/283
17.2/ 30.9
novel
18.6 /
25.1
28.7
/ /1.0
18.9
25.8 /8.9/25.8/
30.0
16.6/
25.2 31.5
attention, we observe consistent and substantial improvements for both R @ K and mR @K on both base and novel split. These results demonstrate the efficacy of our adapter and necessity of incorporating cross-attention for capturing the complicated interplay between subjects and objects. Since DM designed for geometrie relations with massive annotations [21, 24], the improvements on R @ K are considerable, while those mR @ K are relatively small. See $A for more results.
5 Conclusion
This work presents SDSGG, a scene-specific description based framework for OVSGG. Despite the previous works based upon calegory-Ipart-level prompts, argue that these text classifiers are

---

scene-agnostic, which cannot adapt to specific contexts and may even be misleading. To address this. we carefully design a multi perstin collaboration strategy for generating flexible, context-aware SSDs, self-normalized similarity computation module for renormalizing the influence of each SSD, and a mutual visual adapter that consists of several trainable lightweight modules for learning interaction-aa aware space. Our approach distinguishes itself by using SSDs derived from LLMs, which are tailored to the content of the presented image. This is further enhanced by MVA, which captures the underlying interaction patterns based on the semantic space of VLMs. We expect the introduction of SDSGG will not only set a new benchmark for OVSGG, but also encourage the community to explore the potential of integrating LLMs with VLMs for deeper, contextual understanding of images.
Acknowledgement. This work was supported by the National Science and Technology Major Project (No. 20232D0121300), the National Natural Science Foundation of China (No. 62372405), the Fundamental Research Funds for the Central Universities 226-2024-00058, National Key Laboratory of Human-Maching Hybrid Augmented Intelligence, Xi' an Jiaotong University (No. HMHAI- 202403), Bytedance Doubao Fund, and the Earth System Big Data Platform of the School of Earth Sciences, Zhejiang University.
References
[1] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, pages 5410-5419, 2017.
[2] Tao Hc, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Towards open-vocabulary seene graph generation with crompt-based finetuning. In ECCV, pages 56-73, 2022.
[3] Qifan Yu, Juncheng Li, Yu Wu, Siliang Tang. Wei Ji, and Yueting Zhuang. Visually-prompter language model for gra-irained scene graph generation in open world. In ICCV, 2023.
[4] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML. pages 8748-8763, 2021.
[5] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICMI,, pages 12888-12900, 2022.
[6] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys, 55(9):1-35, 2023.
[7] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 130(9))2337-233 2248,, 2022.
[8] Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. Compositional prompt tuning with motion cues for open- apen-vocabulary video relation detection. In ICLR, 2023.
[9] Yong Zhang, Yingwei Pan, Ting Yao, Rui Huang, Tao Mci, and Chang Wen Chen. Learning to generate scene graph using pre-trained visual-semantic space. In CVPR,
supervse and open-vocabune pages
2023.
[10] Zuyao Chen, Jinlin Wu, Zhen Lei, Zhaoxiang Zhang, and Changwen Chen. Expanding scene graph boundaries: Fully open- open-vocabulary scene graph generation via visual-concept alignment and retention. arXiv preprint arXiv:2311.10880 2023.
[11] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
[12] Lin Li, Jun Xiao, Guikun Chen, Jian Shao, Yucting Zhuang, and Long Chen. Zero-shot visual relation detection via composite visual cues from large language models. In NeurIPS, 2023.
[13] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang. Andrej Karpathy, Aditya Khosla, Michael S. Bernstein, Alexander C. Berg, and Fei-Fei Li. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-255 2015.
[14] Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata. Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32-73, 2017.
[15] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, pages 6700-6709, 2019.
[16] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, pages
2018.
10

---

[17] Kaihua Tang, Hanwang Zhang. Baoyuan Wu, Wenhan Luo, and Wei Liu. Learning to compose dynamic tree structures for visual contexts. In CVPR. pages 6619-6628, 2019.
[18] Xin Lin, Changxing Ding, Jinquan Zeng, and Dacheng Tao. Gps-net: Graph property sensing network for scene graph generation. In CVPR, pages 3746-3753, 2020.
[19] Xin Lin, Changxing Ding, Jing Zhang, Yibing Zhan, and Dacheng Tao. Ru-net: Regularized unrolling
network for scene graph generation. In CVPR, pages 19457-19466, 2022.
[20] Chaofan Zheng, Xinyu Lyu, Lianli Gao, Bo Dai, and Jingkuan Song. Prototype-base embedding network for scene graph generation. In CVPR, pages 22783 22012222222202 2023.
[21] Kaihua Tang, Yulei Niu, Jianqiang Huang. Jiaxin Shi, and Hanwang Zhang. Unhiased scene graph generation from biased training. In CVPR, pages 3716-3725, 2020.
[22] Lin Li, Guikun Chen, Jun Xiao, Yi Yang, Chunping Wang, and Long Chen. Compositional feature augmentation for unbiased scene graph generation. In ICCV, pages (1685-21694 2023.
[23] Lin Jun Xiao, Hanrong Shi, Wenxiao Wang, Jian Shao, An-An Liu, Yi Yang. and Long Chen. Label semantic knowledge distillation for unbiased scene graph generation. IEEF TCSVT, 2023.
[24] Lin Li, Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang. and Jun Xiao. The devil is in the labels: Noisy label correction for robust seene graph generation. In CVPR, pages 18869-18878, 2022.
[25] Xingchen Li, Long Chen, Jian Shao, Shaoning Xiao, Songyang Zhang, and Jun Xiao. Rethinking the evaluation of unbiased scene graph generation. In BMVC, 2023.
[26] Guikun Chen, Lin Li. Yawei Luo, and Jun Xiao. Addressing predicate overlap in scene graph generation with semantic granularity controller. In ICME, 2023.
[27] Lin Li, Jun Xiao, Hanrong Shi, Hanwang Zhang, Yi Yang, Wei Liu, and Long Chen. Nicest: Noisy label correction and training for robust scene graph generation. IEEE TPAMI, 2024.
[28] Hanrong Shi, Lin Li, Jun Xiao, Yueting Zhuang, and Long Chen. From easy to hard: Learning curricular shape-aware features for robust panoptic scene graph generation. IJCV. pages 1-20, 2024.
[29] Jing Shi, Yiwu Zhong, Ning Xu, Yin Li, and Chenliang Xu. A simple baseline for weakly-superviset scene graph gencration. In ICCV, pages 16393- 2021.
[30] Xingehen Li, Long Chen, Wenbo Ma, Yi Yang, and Jun Xiao. Integrating object-aware and interaction- aware knowledge for weakly supervised scene graph generation. InACM MM. pages 2204-4213, 2022.
[31] Alireza Zarcian, Svebor Karaman, and Shih-Fu Chang. Wcakly supervised visual semantic parsing. In CVPR, pages 3736-3745, 2020.
[32] Keren Ye and Adriana Kovashka. Linguistic structures as weak supervision for visual scene graph 2021.
generation. In CVPR, pages
[33] Yuan Yao, Ao Zhang, Xu Han, Mengdi Li, Cornelius Weber, Zhiyuan Liu, Stefan Wermter, and Maosong Sun. Visual distant supervision for scene graph generation. In ICCV, pages 15816-15826, 2021.
[34] Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Toward a unified transformer-based framework for scene graph generation and human-object interaction detection. IEEE TIP, 32:6274-6288, 2023.
[35] Yao Teng and Limin Wang, Structured sparse r-enn for direct scene graph generation. In CVPR, pages 19437-19446, 2022.
[36] Yuren Cong. Michael Ying Yang, and Bodo Rosenhahn. Reltr: Relation transformer for scene graph generation. JEEE TPAMI, 2023.
[37] Hengyue Liu, Ning Yan, Masood Mortazavi, and Bir Bhanu. Fully convolutional scene graph generation. In CVPR, pages 1 11546-11558 11556, 2021.
[38] Rongjic Li, Songyang Zhang, and Xuming He. Sgtr: End-to-end seene graph generation with transformer. In CVPR, pages 19486- 19496, 2022.
[39] Minghan Chen, Guikun Chen, Wenguan Wang, and Yi Yang. Hydra-sgg: Hybrid relation assignment for one-stage scene graph generation. arXiv preprint arXiv:2409.10266 2024.
[40] Liulei Li, Jianan Wei, Wenguan Wang, and Yi Yang. Neural-logic human-object interaction detection. In NewrIPS, 2023.
[41] Liulei Li, Wenguan Wang, and Yi Yang. Human-object interaction detection collaborated with large relation-driver driven diffusion models. In NeurlPS, 2024.
[42] Jianan Wei, Tianfei Zhou, Yi Yang, and Wenguan Wang. Nonverbal interaction detection. In ECCV, 2024.
[43] Xingchen Li, Jun Xiao, Guikun Chen, Yinfu Feng, Yi Yang, An-An Liu, and Long Chen. Decomposed prototype learning for few-shot scene graph generation. ACM TOMM, 2023.
[44] Yongqin Xian, Zeynep Akata, Gaurav Sharma, Quynh Nguyen, Matthias Hein, and Bernt Schiele. Latent embeddings for shot classification. In CVPR, pages 77, 2016.
11

---

[45] Ankan Bansal, Karan Sikka, Gaurav Sharma. Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In ECCV, pages 384-400, 2018.
[46] Bernardino Komera-Paredes and Philip Torr. An embarrassingly simple approach to zero-shot learning. In ICML, pages 2152-2161, 2015.
[47] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-imag pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. /2597, 2023.
[48] Jianzong Wu, Xiangtai Li, Shilin Xu, Haobo Yuan, Henghui Ding, Yibo Yang. Xia Li, Jiangning Zhang, Yunhai Tong, Xudong Jiang, et al. Towards open vocabulary learning: A survey, IEEE TPAMI, 2024,
[49] Xiuye Gu, Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Spen-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.
[50] Jian Ding, Nan Xue, Gui- GuinSong Xia, and Dengxin Dai. Decoupling zero-shot semantic segmentation. In CVPR, pages 11593, 2022.
[51] Size Wu, Wenwei Zhang, Sheng Jin, Wentao Liu, and Chen Change Loy. Aligning bag of regions for open-vocabulary object dotection. In CVPR, pages 15254-15264, 2023.
[52] Le Xue, Mingfei Gao, Chen Xing, Roberto Martin-Martin, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding. In CVPR, pages 1179-1 1189, 2023.
[53] Songyou Peng. Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys. Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR, pages 815-824, 2023.
[54] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In /CLR, 2022.
[55] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang, Open-vocabulary object detection using captions. In CVPR, pages 14393-14402, 2021.
[56] Golnaz Ghiasi, Xiuye Gu, Yin Cui, and Tsung-Yi Yi Lin. Scaling open-vocahulary image segmentation with image-level labels. In ECCV, pages 540-557, 2022.
[57] Runyu Ding, Jihan Yang. Chuhui Xuc, Wenqing Zhang, Song Bai, and Xiaojuan Qi. Pla: Language-driven open-vocabulary 3d scene understanding. In CVPR, pages 7010-7019, 2023.
[58] Dat Huynh, Jason Kuen, Zhe Lin, Jiuxiang Gu and Ehsan Elhamifar. tion via robust cross- cross-modal secudo-labeling In CVPR, pages 7020-7031, 2022.
vocabulayy instance segmenta-
[59] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, pages 696-712, 2022.
[60] Feng Liang. Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculeseu. Open-vocabulary semantic segmentation with mask-adapted clip. In CVPR, pages 7061-7070, 2023.
[61] Mingfei Gao. Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong, Open vocabulary object detection with pseudo bounding- labels. In ECCV, pages 266-282, 2022.
[62] Vibashan VS, Ning Yu, Chen Xing, Can Qin, Mingfei Gao, Juan Carlos Niebles, Vishal M Patel, and Ran Xu. Mask-free ovis: Open-vocabulary instance segmentation without manual mask annotations. In CVPR, pages 23539-23542 2023.
[63] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In CVPR, pages 18134-18144 144, 2022.
[64] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiandong Hc, and Tianrui Li. Segelip: Patch aggregation with lcarnable centers for open-vncahulary semantic segmentation. In ICMI., pages 23033-23044, 2023.
[65] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianwei Yang, and Lei Zhang. simple framework for open-vocabulary segmentation and detection. In ICCV, pages 1020-1031, 2023.
[66] Xueyan Zou, Zi- Yi Dou, Jianwei Yang. Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, et al. Generalized decoding for pixel, image, and language. In CVPR, pages 15116-15127, 2023.
[67] Jie Qin, Jie Wu, Pengxiang Yan, Ming Li, Ren Yuxi, Xuefeng Xiao, Yitong Wang, Rui Wang. Shilei Wen, Xin Pan, et al. Freeseg: Unified, universal and open-vocabulary image segmentation. In CVPR, pages
19446-19455, 2023.
[68] Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. Detecting twenty- thousand classes using image-level supervision. ECCV, pages 350 00000668200 2022.
[69] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open- vocabuuayyl object detection. In NewrIPS, 2024.
12

---

[70] Wenguan Wang, Yi Yang, and Yunhe Pan. Visual knowledge in the big model era: Retrospect and prospect. FITEE, 2024.
[71] Ting Chen, Simon Korblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations, In ICML, pages 1597-1607, 2020.
[72] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised
visual representation learning. In CVPR, pages 9729-9738, 2020.
[73] Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. In NeurlPS, pages 22243-22255, 2020.
[74] Xinlei Chen Saining Xie*, and Kaiming He. An empirical study of training self-supervvsed vision transformers. In ICCV, pages 1920-9629, 2021.
[75] Longlong Jing and Yingli Tian. Self- superviedd visual feature learning with deep neural networks: A survey. IEEE TPAMI, 43(11):4037 4058, 2020.
[76] Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Nichles, and Steven CH Hoi. Align and prompt: aideo-and-language pre-training with entity prompts. In CVPR, pages 4953-4963, 2022.
[77] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongic, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, pages 709-727, 2022.
[78] Muhammad Uzair Khattak, Hamoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shahbaz Khan. Maple: Multi-modal prompt learning. In CVPR, pages 19113-19122, 2023.
[79] Mayug Maniparambil, Chris Vorster, Derek Molloy, Noel Murphy, Kevin McGuinness, and Noel
O' O'Connor. Enhancing clip with gpt-4: Hamessing visual descriptions as prompts. In ICCV, pages 262-271, 2023.
[80] Sarah Pratt, Ian Covert, Rosanne Liu, and Ali Farhadi. What does a platypus look like? generating customized prompts for zero-shot image classification. In ICCV, pages 15691-15701 2023.
[81] Leonard Salewski, Stephan Alaniz, Isabel Rio-Torto, Eric Schulz, and Zeynep Akata. In-context imperson- ation reveals large language models' strengths and biases. In NeurIPS. volume 36, 2024.
[82] Renrui Zhang, Xiangfei Hu, Bohao Li, Siyuan Huang. Hanqiu Deng, Yu Qiao, Peng Gao, and Hongsheng Li. Prompt, generate, then cache: Cascade of foundation models makes strong few-shot learners. In CVPR, pages 15211-15222, 2023.
[83] Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison- Burch,, and Mark Yatskar. Language in a bottle: Language model guided concept bottlenecks for interpretable image classification. In CVPR, pages 19187-19197, 2023.
[84] An Yan, Yu Wang. Yiwu Zhong, Chengyu Dong, Zexue He, Yujie Lu, William Yang Wang, Jingbo Shang, and Julian McAuley. Learning concise and descriptive attributes for visual recognition. In ICCV, pages 3090-3100, 2023.
[85] Lin Li, Guikun Chen, Jun Xiao, and Long Chen. Compositional zero-shot learning via progressive language-based observations. arXiv preprint 2rXiv:2311.1474p 2023.
[86] Zhenhailong Wang, Shaoguang Mao, Wenshan Wu, Tao Ge, Furu Wei, and Heng Ji. Unleashing the emergent cognitive synergy in large language models: task-solving agent through multi-persona self- collaboration. arXiv preprint arXiv:2307.055..0 2023.
[87] Haoyang Liu, Yijiang Li. Jinglin Jian, Yuxuan Cheng, Jianrong Lu, Shuyi Guo, Jinglei Zhu, Mianchen Zhang. Miantong Zhang. and Haohan Wang. Toward a team of ai-made scientists for scientitic discovery from gene expression data. arXiv preprint arXiv:2402. 223,, 2024.
[88] Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He. Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. Large language models are better reasoners with self-verificationn In Findings of EMNLP, pages 2550-2575, 2023.
[89] Yu Shang. Yu Li, Fengli Xu, and Yong Li. Defint: A defaalt-interrvetioonss framework for efficient reasoning with hybrid large language models. arXiv preprint 2ww.:::::202256555 2024.
[90] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[91] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV. 2020.
[92] Weizhe Hua, Zihang Dai, Hanxiao Liu, and Quec Le. Transformer quality in linear time. In ICML, pages 9099-9117, 2022.
[93] Chun-Fu Fu Richard Chen, Quanfu Fan, and Rameswar Panda. Crossvit: transformer for image classification. In ICCV, pages 357-366, 2021.
multi-scale vision
13

---

[94] Wenxiao Wang, Wei Chen, Qibo Qiu, Long Chen, Boxi Wu, Binbin Lin. Xiaofei He. and Wei Liu. Crossformer++: A versatile vision transformer hinging cross-scale attention. IEEE TPAMI, 2023.
[95] Guangyi Chen, Weiran Yao, Xiangchen Song, Xinyue Li, Yongming Rao, and Kun Zhang. Plot: Prompt learning with optimal transport for vision vision-language language models. In ICLR, 2023.
[96] Yaohua Zha, Jinpeng Wang, Tao Dai. Bin Chen, Wang, and Shu-Tao Tao Xia. Instance-aware dynamic prompt tuning for pre-trained point cloud models. In ICCV, pages 14161-14170, 2023.
197] Lifeng Fan, Wenguan Wang. Siyuan Huang, Xinyu Tang, and Song Chun Zhu. Understanding human gaze communication by spatio-temporal graph reasoning. In ICCV, pages 5724-5733, 2019.
[98] Jiaming Lei. Lin Li, Chunping Wang, Jun Xiao, and Long Chen. Seeing beyond classes: Zero-shot grounded situation recognition via language explainer. In ACM MM, 2024.
14

---

Summary of the Appendix
For better understanding of the main paper, we provide additional details in this supplementary material, which is organized as follows:
A introduces more ablative experiments.
BB details the implementation details.
SC provides the pseudo code of SDSGG.
$D shows the generated scene-specific descriptions and the corresponding prompts.
SE offers more qualitative results.
SF discusses our limitations, societal impact, and directions of future work.
SG presents more experimental results.
A More Ablative Experiment
Self-normalized Similarity. Furthermore, we study the effectiveness of self-normalized similarity for evaluating the influence each scene- specific description ($3.1) Table $1. Here we remove scene-specific
the opposite description and use only the original scene description as the baseline. As seen, without the reference of opposite description, the performance drops drastically, e.g., 28.3 28.3% 2222225.../.... 4%% vs. /00%%/3./%//.... R 100 & 12. //6/2.88%/24. base/novel/ssmantti split, respectively. This indicates the importance renormalizing similarity scores, and also validates our hypothesis the absolute value of one single similarity yields only limited insight, while the difference of two similarities provides more information.
14.7% /2..% mR @ 100 on the
Table S1: Ablation studies ($A) on self-normalized similarity (SNS, $3.1).
Method
Split
R 20
R@501
RØ 1001
mR 20
mR@500 501
mR@ 1001
Ours
18.7
26.6
31.6
9.1
12.3
14.7
base
w/o SNS
16.1
23.3
28.3
7.0
10.0
12.1
Ours
18.9
25.8
30.0
16.6
25.2
31.5
novel
w/o SNS
14.7
20.6
25.5
14.2
21.5
29.8
Number of SSDs. We first study the impact of the number of used classifiers (i.e., SSDs). The results, presented in Table S2, reveal a considerable performance enhancement when the number of pairs increased from 11 to 21, affirming the efficiency of SSDs. However, performance when too many SSDs were employed. This could be due to the unnecessary pairing of a relation category with an excessive number of descriptions (52 descriptions for 26 pairs).
also witnessed dip in
Table S2: Ablation studies ($A) the number of used classifiers (Eq. 2).
The adopted hyperparameters
marked in red.
#
Pairs
Split
R 20
R@50
R@ 100
mR@22 20
mR@50
mR@ 100
11
13.9
19.8
24.0
8.6
12.0
14.3
16
19.4
27.5
32.9
8.6
11.8
14.2
base
21
18.7
26.6
31.6
9.1
12.3
14.7
20.0
7.7
18.4
23.9
26.8
16.6
23.3
27.2
17.9
24.4
28.7
16.4
23.2
28.7
novel
21
18.9
25.8
30.0
16.6
25.2
31.5
26
12.4
18.5
23.7
14.3
21.8
27.2
Scaling Factors. We then study the effectiveness of the scaling factors used in our training objectives ($3.3) in Table S3. As seen, the performance remains consistent for those compared hyperparameters. This indicates the robustness of SDSGG to changes in the scale the training targets. Considering the performance all metrics together, and B
to be and le-1, respectively.
Number of Attention Heads. We further investigate the impact the number of attention heads H used in the mutual visual adapter ($3.2, Eq. 5). As shown in Table S4, the performance improves
15

---

Table S3: Ablation studies ($A) the scaling factors used in the training objectives, Le., and B ($3.3, Eq. 8).
adopted hyperparameters are marked red.
Split
R@ 207
R@
R@ 1001
mR@20
mR @ 501
mR @ 1001
1.5
le-1
16.8
24.0
28.7
9.8
12.7
15.0
2.5
le-1
19.4
27.3
32.4
9.4
12.6
14.9
3c-1
18.8
26.7
31.8
9.4
12.6
14.9
base
Se-1
17.2
9.2
12.2
14.4
18.6
26.4
31.7
9.2
12.1
14.3
2
le-1
18.7
26.6
31.6
9.1
12.3
14.7
1.5
Ic-1
19.8
25.5
29.3
17.5
24.5
30.8
2.5
le-1
18.8
25.2
29.8
19.5
25,9
32.1
3c-1
18.4
25.2
29.7
16.5
24.4
30.4
novel
Se-1
18.5
25.0
29.3
15.8
23.1
29.2
18.8
25.5
29.6
17.3
23.6
30.1
le-1
18.9
25.8
30.0
16.6
25.2
31.5
from 32.7% to 35.1% R@ 100 on the semantic split when increasing the number heads from 4 to 8, but the number of parameters steadily increase as the number of heads grows. When increasing the number heads from 8 to 16, we observe some improvements (e.g.. 31.6% to 32.8% R@ 100 on the base split), but also split). Consequently, we set H = as the default to strike an optimal balance between accuracy and computation cost.
performance drops (e.g., 30.0% 29.0% R@ 100 the novel
Table S4: Ablation studies ($A) on the number of attention heads H used in the mutual visual adapter ($3.2,
Eq. 5). The adopted hyperparameters are marked in red.
Param.
Split
R@201
R@
R@ 1001
mR@207
mR 501
mR@ 1001
4.2M
18.9
27.0
32.5
9.1
12.2
14.4
7.1M
base
18.7
31.6
9.1
12.3
14.7
16
12.8M
19.4
27.5
32.8
9.5
12.7
15.1
4
4.2M
18.2
25.3
29.9
15.9
24.8
31.0
8
7.1M
novel
18.9
25.8
30.0
16.6
25.2
31.5
16
12.8M
18.0
25.3
29.0
15.8
24.0
29.1
Margin. Last, examine the impact of the margin À in training objectives ($3.3, Eq. 8). As shown in Table S5, the performance remains consistent for those compared values, which indicates the robustness of our learning procedure. À is set to be 3e-2 by default.
Table S5:
Ablation studies ($A) on margin À (Eq. 8).
The adopted hyperparameters are marked red.
Split
R@20
R@50
R 100
mR@20
mk @50
mk @ 100
5c-2
19.2
26.9
32.0
9.5
12.6
15.0
3e-2
base
18.7
26.6
31.6
9.1
12.3
14.7
lc-2
18.6
26.0
30.8
8.8
11.8
13.8
5c-2
18.7
24.8
29.1
17.0
24.0
30.3
3c-2
novel
18.9
25.8
30.0
16.6
25.2
31.5
le-2
18.0
24.6
28.7
16.6
24.4
29.2
B Implementation Details
Training. Our model is trained with batch size of 4. One RTX 3090 used for training. During the training process, images are resized to dimensions within the range of [600, 1,000]. For each relation, up to 50K samples are included. Random flipping is adopted for data augmentation. SGD adopted for optimization. The initial learning rate, momentum, and weight decay are set to be 2e-2, 9e-1, le-4, respectively. We utilize the pre-trained weights of CL.IP to initialize our model. avoid data leakage, remove annotations in the training set which contains categories in the novel split.
16

---

Testing, Testing is conducted on the same machine as in training. No data augmentation is used during testing. terms of the semantic split, we directly applied the same weights trained on the base split for testing. A similar filtering strategy [12] is adopted.
Codebase and Architecture. We use the same codebase in [21]. We adopt GPT-3.5 from OpenAl as our LLM. In terms of CLIP, we employ a widely used architecture, i.e., ViT-B/32, for initializing our mutual visual adapter.
Split. We visualize all relation categories in Table S6. To guarantee fairness, we select GQA's [15] relation categories that also exist in VG [14]. As such, on VG, thus verifying the generalizability of the proposed SSDs and self- normalizing similarity.
reuse SSDs obtained in our experiments
Table S6: Relation categories in each split.
Split
Relation Calegories
watching, of, hanging from, lo, near, carrying. parked on, covered wearing. from, in front of, helonging between, above, attached walking on. behind, in, holding. against, has. looking at, under, at. playing. riding, covering, for, with, wears, over
sitting made of, on, standing
Base
flying in, painted on. mounted on, using, and. on back of. growing on, lying on, along, part of, eating, laying on, walking in, across, says
Novel
watching, growing hanging from, eating, carrying, parked on, covered in, says, using. flying in, painted on, sitting lying on, standing walking in, mounted on, attached to, walking on, holding, looking at, playing, riding, covering, laying on
Semantic
(a) VG [14]
Sphit
Relation Calegoress
in front of, watching, riding, covered in, under, wearing. behind, parked on, covering, above, of, on, holding, walking on, al, carrying, with, sitting on, in, standing on, near
Base
lying on, looking al, growing on, walking in, attached using, mounted hanging from, flying in, cating
Novel
(b) GQA [15]
C Pseudo Code
The pseudo code of SDSGG is given in Algorithm S1 and Algorithm S2. We respectfully refer the reviewer to the supplementary Python files for the PyTorch implementation of SDSGG's key modules. Moreover, to guarantee reproducibility, our full code and pre-trained models will be publicly available.
Algorithm S1 Pseudo-code for MVA of SDSGG in a PyTorch-like style.
fegtfe subject or object visuel features, where the first vector is CLS feature, and the left vectors are patch features.
ing_fosts2: subject object visual features. The format is the ing_featsl. dira_mark_ narker used idantity the diraction the relation.
def NVA fing.feae,,,, feats2, dire_mark_feat)) Project features into loo-dinensional, interation-aurre space. Eq.
fatch_ = (onnarDoon(ing.foatallii
Hodal the interplay batueen subject object out (patstttaaaaccc_feaa ing_feats2[1:]]) # Eq- 5
Project features into original dinension. out Linearl)p(ont) Eq.
Add direction ava_out Linear (Cut, dire mark feature))
mva_out (ava_out + ing_foatsi [o])s((]) m) # Eq. 6 return
17

---

Algorithm S2 Pseudo-code for the forward process of SDSGG in PyTorch-like style.
bounding
objects.
ing: input inage.
text_raw, rew_ees: description.
text_op_dde ops opposite description.
as80: the association presented in "Association betveen Scene-level Deacriptions and Relation Categories".
targete: training targets. nea
def (bbores,
text_rav_dee, text_op_dee, targete):
Batch crop and ancoda inages. crop_ing Crop(ing, bboxes)
feats EncodeInage ingo CLIP's encoder, Eq.
Tokenize and encede descriptive text.
CLIP's ancoder text_op__feate EncodeText (Toeenizeeteett.ppp.
EncodeText (ttrrttsttttttrraaa..o
Dafine diractional
aark.aub (Tokeeiiie photo of aubject"))
mark_obj (Toxeniizee photo of object"))
acorea, loases
Calculate self-normalized similarity for each subject-object pair for sub, obj sub_obj.paire:
feat[ ing_feats mark_aub) o2e_ XVV Winni reats[ob]], ing_ [subl, nbrk_obj) ava_out (mva_out__22 ava_out_o2s))
ava put_
sin_raw CosSin(ava_orr text_rax_Ieetta)
ein_ CosSin(nva_out, text_op_feate)
score ((sin_rav ain_op) asso) sun() # Eq. 2
ecoree appree.epen
If is training training:
loss compute_ loaa(sin__ar,, sim_op, targets) # Eq. aossee.a
return acores, lossea
18

---

D Scene-specific Description
The detailed, scene-specific descriptions generated by LLM's muulti-persone collaboration are provided in Fig. S1. As seen, our scene- -specifc descriptions (21 pairs, 42 descriptions in total) cover different scenes from comprehensive view, e.g., spatial descriptions ("two or more objects partially overlap each other", "interaction between objects", etc.), environment (background) descriptions ("on road", "on a flat plane, it should appear balanced with no visible tilting". etc.), semantic descriptions ("belong to animal or human behavior". "may have contact behavior", etc.), appearance descriptions ("might have flat teeth or sharp teeth", "have a curvy body", etc.).
Have curry body
of more objectts partully ovarlap each other Each object is compleslly separate with clear space between
Hawe stright body
Generslly used decomation
Not inter dud for deconative purposes
Intenctioon betaeeen objects
each ocher.
Chyect mwmiins diationary
picture amother ctject
Dowlspment
With the dicture not being on part the object
M4g hove teeth seath
Ortects are poetioned hortonbally in rebetton to
rosembllng
So soltt material
On road
Campored hand, rigit meteetl
SSDs
Located off-rad, in non-mad invironment
Posture ind cacing concentration
May have co bolf owkor
Posture ind racive of distraction disinteeeet or ie loloation
They canset touch or internet
On that plans, should appear balanced with
object innoress
Sonsery organs is are directed away are Bellong to antrmal or human behavier
even surface appearing
imbollance
Special la leed structures resembling legs
Possilly Inaniiaas or medianical
Lackss strucc resembbting logs
Have specialied equioment
With hand, for grapping support, locometion
Lacks any form of seeciateed equipment, entiney unaided or basic form
Lacks hands simler structures
Connected attached
structure
Stuated an span. unrsetricted envrrommnn
larger atructure
Figure S1: Illustration of the generated scene-specific descriptions.
Next, we detail each step to generate scene-specific descriptions.
O Initial Description Generation. This step can be repeated many times to generate a large number (72 after manual selection) of scene descriptions. The prompt shown in Fig. S2.
Summarizing Descriptions. Since these initial descriptions may suffer from noise and semantic overlap, we ask LLM to streamline and combine these descriptions, ensuring more cohesive and distinct scene-level descriptions. The prompt is shown in Fig. S3.
@ Enacriptiio-Relatiion Association. After obtaining various scene descriptions, a critical inquiry arises regarding their utility for relation detection, given their lack of explicit association with specific relation categories. To address this, we delineate three distinct scenarios characterizing the interplay between relation categories and scene descriptions: i) certain coexistence (CTL 1), where a direct correlation exists; ii) possible coexistence (C = 0), indicating a potential but not guaranteed association; and iii) contradiction (C"- = -11, 1), denoting an incompatibility between the scene description and relation category. Here Ct denotes the correlation between relation E R and Nsh scene description, and is generated by LLMs. The prompt is shown in Fig. S4.
@ Opposite Description Generation. Since classifiers are contextually bound, we generate opposite descriptions to compute self-normalized similarities. The prompt is shown in Fig. $5.
19

---

Input: (scene content be discussed}
Output: 3 descriptions
Begin by embodying three distinct personas: biology expert, physics expert, and engineering expert. Each expert will articulate step-by-step approach along with their thought proceas, considering various hypothetical scenarios relevant to their field of study, and then share their ineights with the group. If any expert does not know the answer, he will exit the discussion. Once all experts have provided their analyses, summarize the final generic descriptions.
The generic scene description involves the specific appearance
visible may appear in the scene (use your imagination here).
Here are some examples of generic scene descriptions: {some in-context examples)
Here example of discuasion:
Discussion started!
Question: Suppose there is (this involves a detailed discussion of three roles)
Discussion started!
Question: Suppose there {acene content discussed), please give concise, generic scene descriptions this scene.
Figure S2: Prompts for initial description generation.
Input: initial (scene descriptions), all frelation categories), {number final scene descriptions}
# Output: final acene-level descriptions
Here is text pool that includes series of descriptive texts: (scene descriptions) Here are all the relation categories:
(relation categories}
You are asked to pick (number of final scene descriptions) descriptive statements from the pool that describe least predicates. Think step by step.
Figure S3: Prompts for summarizing descriptions.
Input: (scene descriptiona), all frelation categories] Output: description-relate associations
For (scene descriptions), decide whether it is likely to appear in one of the following flat photos where (relation categories} appears.
The judgment reault is choose between [certain coexistence, possible coexistence, and contradiction). When the photo shows that the scene must have a certain description, it is judged as "contradiction". When the photo has little relationship with the description, it is judged as "possible must not appear with
coexistence", When the photo shows that the certain description, is judged as "contradiction".
Figure S4: Prompts for generating descriptiin-relltioo associations.
20

---

# Input: (scene descriptiona) (original scene descriptions) Output: final scene-level descriptions nan
Here are some descriptions:
(scene descriptions)
Please give their opposite descriptions.
Figure S5: Prompts for opposite description generation.
E More Qualitative Comparison Result
We provide more visual results that compare SDSGG to CLIP [4] in Fig. S6. It can be observed that SDSGG performs robust in hard cases and can consistently deliver more satisfying results, based upon the scene-specific descriptions and self-normalized similarities.
CLIP
walking
ridngg
20.5%
walking
225%
bying
sanding
belding
looking
onsceed
walking
192%
carging
11.3%
walking
bolding
Spoople, walking on,
Stadling
Sman, currying, bagó
R.%
ne reticioz
23.5
langing from
20.0
looking
starding
holding
2875
holding
SUSGG
SDSGG
riding
Staddnn on
<man, niding,
13.2%
13.8%
Omin, standing 600
walling
11.55
a stedl envinemenn A.
specialred tructs rea loge-
Siuated in unmariicted enaloonmert
delta 0.20
Lade eosecaee reseridd agp
vericcll postional relatinntip aun
delta 0.20
Lecda form sisedell and eqpioment
vnlelraste and
delta
-elta-00
Hame ca de
Cipyct tonvars stasianrry Add
shraght
Payture batinng concentotionn 2.00
Purtare rectae intcatte of sour on.
remaiin rustonary
riding
delta 188
standing
delta =171
welongg andl betaeior
Trey cenner south frowece 236
Teen rone régers contaily rven
delta - 176
weang aninall feance betuider
ba iuriimate meder siciaal
detta 1154
Eat buclutent cocrediit vos
Two marcotedds oily over each
delta -
Cach oblett compleslly aepursee
delta - 134
objec coompiene
spaca betioeen
delta
delta
Figure S6: Visual results ($E) on VG [14] test. As seen, SDSGG makes the right predictions based upon the scene-specific descriptions and self-normalized similarities.
21

---

Discussion
Limitation Analysis. Following [12], currently our algorithm specifically designed for the predicate classification task [1, 16] to make fair comparisons with relation detection models. Such a pairwise classification task gives the ground-truth annotations of objects in the scene for easier representation learning of relations. However, these annotations unavailable in real-world applications.
Societal Impact. This work points out the drawbacks of existing scene-agnostic text classifiers and accordingly introduces several new modules for both the visual and textual components, leading to scene-specific description based OVSGG framework that combines the strengths of VLMs and LLMs. Like every coin has two sides, using framework will have both positive and negative impacts. On the positive side, our work contributes to research on intelligent scene understanding. and thus is expected to eventually benefit industries such autonomous driving. For potential negative social impact, the reliance on VLMs and LLMs could lead to the perpetuation of biases and inequalities present in the data used in these models' large-scale pre-training stage.
Future Work. As mentioned before, the focus this work is not on object detection. is interesting to extend our algorithm to handle the object detection task simultaneously by, for example, incorporat- ing set-prediction architectures [91]. Moreover, the design of our multi-persona collaboration stands for early attempt and deserves to be further explored. In addition, the architectural designs of directional marker and mutual visual adapter certainly worth further explorations, e.g., efficiency [92], architecture [93, 94], and adaptive prompting [95, 96]. Furthermore, extending our algorithm to other relation detection tasks 197, 40, 42, 98] may lead to an uniformed relation detection algorithm.
G More Experiment
Training on the Full Set of Relation. We trained our model with frequency bias [25] the full set of relations. The results are shown in Table S7.
Table Results on full set of relation.
Method
mR 50@501
mR @ 100
Ours
28.7
34.2
Different Base/Novel Splits. We trained our model on different base/novel splits to investigate the robustness further. Specifically, we i) change the proportion of the base and novel split and ii) change the categories within the base and novel split (i.e., different No. for the same ratio). The results are shown in Table S8.
Table S8: Results on different base/novel splits.
No.
base:novel
Base
Novel
mR 50
mR @ 100
mR @ 50
mR 100
(paper)
35:15
12.3
14.7
25.2
31.5
2
35:15
12.4
14.8
24.3
28.2
3
32:18
11.9
14.4
23.9
28.4
32:18
13.6
15.9
20.6
26.7
38:12
11.8
14.2
23.7
29.9
38:12
11.5
13.7
22.6
27.1
Inference Time. Since the renormalization and similarity measurement involve only few matrix operations that can be omitted from the complexity analysis, we will focus on the inference time of three main modules. The results are shown in Table $9.
Table $9: Inference time for different modules.
Module
Inference Time (ms/pair)
CLIP's Visual Encoder
6.5
Mutual Visual Adapter
0.2
CLIP's Text Encoder
5,4
22

---

Different Personas. The involvement of multiple personas in the discussion process enhances the diversity of generated descriptions. The key point of our multi-persona collaboration is about the "collaboration" rather than specific persona. Actually, using only one persona can even decrease the diversity of generated descriptions and hurt the performance, as it can only generate descriptions from its own viewpoint without discussion with others. In addition, we changed the system prompt of LLM from the default like "you are a helpful AI assistant" into persona-specific one like "you a biologist". We then evaluate the performance of our model with these generated descriptions. The results are shown Table S10.
Table S10: Comparison of different personas.
Method
Base
Novel
mR@ 501
mR @ 1001
mR@501
mR 1001
Multi-persona Collaboration
12.3
14.7
25.2
31.5
Biologist Persona
6.3
8.4
12.9
18.4
Engineer Persona
7.3
9.7
8.4
11.5
Physicist Persona
4.8
6.9
9.1
14.7
23

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We carefully described our contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the appendix, we discussed our limitations, societal impact, and directions for future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]

---

Justification:
This paper is not about theory.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provided details about the methodology and implementation in the main paper and appendix. The code will be publicly available.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
25

---

Answer: [Yes]
Justification:
The code will be publicly available at https//giihibb.ooonggiiuuuuhhhch
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppcccaeccccccccccccccat / ssssssooiccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We present the experimental setup and details in the main paper and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: deviation.
We run each experiment three times and report the average and standard
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
26

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We introduce the used computer resources in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https : http///eerrpp..eeee
Answer: [Yes]
Justification:
We carefully reviewed the NeurlPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the appendix, we discussed our limitations, societal impact, and directions for future work.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited related papers.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "qro8MwccwF", "reviewer": "Reviewer_qa1m", "summary": "This paper starts by discussing methods for Open-vocabulary Scene Graph Generation (OVSGG) based on the CLIP model, highlighting the issue that current OVSGG methods do not differentiate between various scenes, which limits their effectiveness. The authors introduce SDSGG, a scene-specific description-based OVSGG framework that improves both the textual and visual parts, enhancing the model's open-vocabulary relationship prediction capabilities.", "questions": "1. Regarding the selection of multiple personas, the ablation study shows that not using this approach results in a significant performance decrease. My question is, what exactly are the \"standard prompts\" referred to in line 329 of the document? What would be the effect if only one persona is used, and among the three personas mentioned in the document, which persona demonstrates the most significant performance?", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The novelty of this paper lies in its analysis of the issues present in current OVSGG methods, leading to the conclusion that differentiating between scenes is necessary to enhance the performance of OVSGG. The proposed Scene-specific Descriptions are particularly insightful.\n2. The paper validates its findings on two datasets, VG and GQA, with experimental results showing significant performance improvements over previous state-of-the-art methods.", "weaknesses": "1. The description in Sec3.1, Scene-specific Text Classifiers, of the paper is somewhat confusing. This confusion arises primarily because the text section includes multiple different naming conventions and several distinct modules. It is recommended that this section be rewritten to make it easier for readers to understand. Additionally, the terminology used in this section is inconsistent with that in lines 64~77, leading to comprehension difficulties.\n2. For the OVSGG method, it is suggested to also train the model on a full set of relations and compare its performance with conventional SGG methods to ensure that it achieves good performance under standard settings.\n3. Is the model robust to different base/novel splits? It is recommended to train and test the model on different base/novel dataset divisions to assess its robustness.\n4. It is advised to train and test the model on the PSG dataset as well.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer qa1m for the valuable time and constructive feedback. We provide point-to-point response below.\n\n**Q1**: **Presentation.**\n\n**A1**: Our apologies. We will revise Section 3.1 to improve clarity and coherence. Our revisions will focus on:\n\n1. Streamlining the naming conventions: We will eliminate the term \"text classifiers\" and consistently use \"descriptions\" throughout.\n2. Delineating modules: We will add a summary of the newly-involved modules and their utility for description generation.\n3. Ensuring consistency: We will standardize the terminology across the manuscript, aligning it with the definitions provided in L64~77.\n\n**Q2**: **Comparison with conventional SGG methods.**\n\n**A2**: Thanks for your suggestion. Per your request, we trained our model with frequency bias on the full set of relations. The results are as follows:\n\n| Method | mR@50 | mR@100 |\n| --- | --- | --- |\n| Motifs$_{CVPR'18}$ | 14.9 | 16.3 |\n| VCTree$_{CVPR'20}$ | 16.7 | 17.9 |\n| RU-Net$_{CVPR'22}$ | - | 24.2 |\n| PE-Net(P)$_{CVPR'23}$ | 23.1 | 25.4 |\n| VETO$_{ICCV'23}$ | 22.8 | 24.7 |\n| DRM$_{CVPR'24}$ | 23.3 | 25.6 |\n| Ours | 28.7 | 34.2 |\n\nAs seen, **our model also demonstrates good performance under standard settings**. We will incorporate the above results into the appendix of our revision. Thanks.\n\n**Q3**: **Experiments on different base/novel splits.**\n\n**A3**: Good suggestion! As you noticed, the mean and variance are reported in Tables 1-3 to ensure the performance advantage is reliable. Per your request, we trained our model on different base/novel splits to investigate the robustness further. Specifically, we 1) change the proportion of the base and novel split and 2) change the categories within the base and novel split (*i.e.*, different No. for the same ratio).  The results are as follows:\n\n|  |  | Base |  | Novel |  |\n| --- | --- | --- | --- | --- | --- |\n| No. | base:novel | mR @50 | mR @100 | mR @50 | mR @100 |\n| 1 (paper) | 35:15 | 12.3 | 14.7 | 25.2 | 31.5 |\n| 2  | 35:15 | 12.4 | 14.8 | 24.3 | 28.2 |\n| 3 | 32:18 | 11.9 | 14.4 | 23.9 | 28.4 |\n| 4 | 32:18 | 13.6 | 15.9 | 20.6 | 26.7 |\n| 5 | 38:12 | 11.8 | 14.2 | 23.7 | 29.9 |\n| 6 | 38:12 | 11.5 | 13.7 | 22.6 | 27.1 |\n\nAs seen, **our model is robust to different base/novel splits.** We will incorporate the above results into the appendix of our revision. Thank you.\n\n**Q4**: **Experiments on panoptic scene graph generation.**\n\n**A4**: Thanks for your suggestion. Our current framework is not directly applicable to PSG. Due to time constraints, we are unable to provide empirical results on the PSG dataset, as significant modifications and engineering efforts are needed. Nonetheless, we recognize the importance of exploring this direction and will definitely consider this as our future work. Thanks.\n\n**Q5: Standard prompts in L329.**\n\n**A5**: Given a *subject-predicate* or *predicate-object* pair, we ask LLM to generate corresponding descriptions. The prompt is defined as:\n\n> Imagine [*subject-predicate* / *predicate-object* pair]. Think about what the scene should look like. Summarize each descriptive statement of the scene in about 15 words each.\n> \n\nWe respectfully refer the reviewer to {reviewer xMLQ Q1} for the rule of prompt construction.\n\n**Q6**: **Clarifying multi-persona collaboration.**\n\n**A6**: As mentioned in L336, the involvement of multiple personas in the **discussion process** enhances the diversity of generated descriptions. The key point of our multi-persona collaboration is about the “**collaboration**” rather than a specific persona. Actually, using only one persona can even decrease the diversity of generated descriptions and hurt the performance, as it can only generate descriptions from its own viewpoint without discussion with others. From the perspective of the number of generated descriptions, each persona's contribution is almost the same.\n\nIn addition, we use the standard prompts and change the system prompt of LLM from the default like “you are a helpful AI assistant” into a persona-specific one like “you are a biologist”. We then evaluate the performance of our model with these generated descriptions. We only report the effect of the biologist and engineer persona because of the urgent due. The results are as follows:\n\n| Method | Base |  | Novel |  |\n| --- | --- | --- | --- | --- |\n|  | mR @50 | mR @100 | mR @50 | mR @100 |\n| Multi-pensona Collaboration | 12.3 | 14.7 | 25.2 | 31.5 |\n| Biologist Persona | 6.3 | 8.4 | 12.9 | 18.4 |\n| Engineer Persona | 7.3 | 9.7 | 8.4 | 11.5 |\n| Physicist Persona |  |  |  |  |\n\nAs seen, using only one persona results in a significant performance decrease, which is consistent with the findings in Table 4. We will give the results of the physicist persona *asap*. Thank you.\n\nReferences:\n\n[a] Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models, NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3IH1ieRl2V", "reviewer": "Reviewer_RK4c", "summary": "This paper aims to solve the open-vocabulary scene graph generation problem. Previous methods mainly adopt scene-agnostic prompts as text classifiers. The authors argue that using the fixed text classifiers not only struggles to model visual relations with high variance, but also falls short in adapting to distinct contexts. Therefore, the authors propose the scene-specific description based OVSGG framework. They employ an LLM and ask it to play different roles. Besides, they design the mutual visual adapter to encode visual features. Extensive experiments show that the proposed method significantly outperforms top-leading methods.", "questions": "Please read the weaknesses part.", "limitations": "Yes", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The motivation and idea of this paper are innovative and interesting. Simply applying LLM to SGG cannot effectively reason the relationships. The authors consider employing the context and introducing multiple roles of LLM, which is shown to be effective for solving the OVSGG problem.\n\nBesides, the experiments are convincing. Plenty of ablation studies are provided.", "weaknesses": "My main concern is Computational Complexity: The proposed framework involves multiple stages, including generating descriptions, renormalizing them, and applying mutual visual adapters. This multi-step process could be computationally intensive, making it less practical for real-time applications or scenarios with limited computational resources.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer RK4c for the valuable time and constructive feedback. We provide point-to-point response below.\n\n**Q1**: **Computational complexity of description generation (offline).**\n\n**A1**: Suppose there are 3 common object categories (*i.e.*, human, animal, and product [a,b]) and 50 predicate categories. The description generation process (L653-667) involves the following steps:\n\n1. Initial Description Generation. This step calls OpenAI's API 2 * 3 * 50 times.\n2. Summarizing Descriptions. This step calls OpenAI's API 1 time.\n3. Description-Relation Association. This step calls OpenAI's API 50 times.\n4. Opposite Description Generation. This step calls OpenAI's API 1 time.\n\nThe execution time of the above steps depends on the network and the speed of OpenAI's response. Since these descriptions are generated offline, **they DO NOT incur any computational cost during deployment.** We respectfully refer the reviewer to {reviewer xMLQ Q2} for the scalability and cost of generating descriptions.\n\n**Q2**: **Computational complexity of model (online).**\n\n**A2**: Since the renormalization and similarity measurement (Eq. 2) involve only a few matrix operations that can be omitted from the complexity analysis, we will focus on reporting the inference time of the following three main modules:\n\n| Module | Inference Time (ms) |\n| --- | --- |\n| CLIP’s Visual Encoder | 6.5 |\n| Mutual Visual Adapter | 0.2 |\n| CLIP’s Text Encoder | 5.4 |\n\nAs seen, the inference time of CLIP’s visual and text encoder is significantly higher than that of our mutual visual adapter. Compared to [b] which also uses CLIP, the delay of the newly-introduced MVA (0.2ms) is neglectable. In addition, it is important to note that during deployment, **only the visual part** (*i.e.*, CLIP’s visual encoder and our mutual visual adapter) **requires computational resources.** This is because the descriptions are generated offline for all categories and remain unchanged during deployment so that their text embedding can be pre-computed and stored. Therefore, the overall framework can meet the requirement of real-time applications or resource-limited scenarios. We will incorporate the results about computational complexity into the appendix of the revision.\n\nReference:\n\n[a] Unbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network, AAAI 2023.\n\n[b] Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models, NeurIPS 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LYf4k5zxgB", "reviewer": "Reviewer_xMLQ", "summary": "This paper proposes SDSGG, a novel open vocabulary scene graph generation(OVSGG) algorithm that leverages the reasoning capability of a LLM to better determine the relations between objects in the scene. It achieves this goal by first prompting a LLM with multiple persona prompts to expand a simple relational predicative to a list of detailed visual descriptions, which are subsequently used to augment the classification process.  It also introduces a novel mutual visual adapter, which better captures the interaction between subjects and objects.  Experiments show that these proposed designs are effective.", "questions": "See weakness.", "limitations": "The authors discussed limitations and societal impact.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Incorporating a LLM to augment the predicate labels for scene graph generation is a novel idea.  This paper provides meaningful insight to future works in this area.\n2. The experiment results (table 1-2) are strong, significantly outperforming previous methods.\n3. The authors conducted extensive ablation studies on various design elements.", "weaknesses": "1. Prompting the LLM is a key element of the method, however some crucial details are missing. For example, how are the prompts constructed? While the author provided the prompt in Appendix Fig 5, it is unclear how the  \"{scene content to be discussed}\" is generated. The author did show some examples throughout the paper, but they are not sufficient for the reader to understand the underlying process. In particular, in L167, the author showed example #1 \"Imagine there is an animal that is eating, \". In Fig 1c, there is example #2 \"Assuming that the scene has a man riding a horse.\"  These two descriptions have two different granularity, as one only includes the generic concept of \"an animal that is eating\" while the other has specific class names \"man\" and \"horse\". The authors should clearly describe what information is included into the prompt, and discuss the scalability and cost of generating such prompts. I suppose if the prompts are like example #1, they can be generated offline based on predicative label sets. However, if the prompts are like example #2, they need to be generated for every possible triple of (subject, predicative, object) over the label space, or be generated online over possible objects in a scene. It is unclear which is the case.\n\n2. Additional discussions and experiments are required to justify some of the design choices. For example,\n\n  2.1 in eq 8, the loss of descriptions marked by possible coexistence is to make the prediction \"close to those of CLIP.\" (L255). If this is the case, why not directly use CLIP results for these possible coexistence descriptions at inference time (eq 2)?\n\n  2.2 some discussion is needed on if CLIP is good at classifying the generated descriptions. What are the nature of these descriptions and do they fit well with CLIP's pretraining pipeline (i.e. object-level image caption)? As a concrete example, can CLIP properly distinguish descriptions involving counting, such as \"with four legs\", and \"with two legs\", mentioned in the examples?\n\n 2.3 what happens if we discard \"possible coexistence\" descriptions and only use definite coexistence and contradiction? Table8 shows that it is ideal to have a low weight for \"possible coexistence\"  loss. What happens if we set the weight to 0 and remove it at inference pipeline?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer xMLQ for the valuable time and constructive feedback. We provide point-to-point response below.\n\n**Q1**: **Clarifying the rule of prompt construction.**\n\n**A1**: The prompts **are like example #1** and **are generated offline**. The \"{scene content to be discussed}\" is constructed based on a given *subject-predicate* or *predicate-object* pair. For instance, in L167 (example #1), the \"{scene content to be discussed}\" is constructed based on a *subject-predicate* pair where the subject is “animal” and the predicate is “eating”. Details about the scalability and cost of generating descriptions are provided in Q2.\n\nRegarding Fig. 1c, it is indeed misleading. In Fig. 1c, we specify a triplet instead of a pair for easier understanding of the later discussion process.  We appreciate the keen observations and recognize that this creates an inconsistency. We will revise Fig. 1c for clarity. The revised version is shown in the PDF (Fig._R1). Thank you.\n\n**Q2**: **Scalability and cost of generating descriptions.**\n\n**A2**: As you noticed, iterating all possible triples of <subject, predicate, object> is **computationally unaffordable** (150 * 50 * 150 for 150 object categories and 50 predicate categories). Therefore, we use *subject-predicate* and *predicate-object* pairs to generate descriptions. Following [a], we use 3 object categories (*i.e*., human, animal, and product) instead of 150 object categories to save computational costs. The description generation process (as mentioned in L653-667) involves the following steps:\n\n1. Initial Description Generation. This step calls OpenAI's API 2 * 3 * 50 times (2 stands for *subject-predicate* and *predicate-object* pairs).\n2. Summarizing Descriptions. This step calls OpenAI's API 1 time.\n3. Description-Relation Association. This step calls OpenAI's API 50 times.\n4. Opposite Description Generation. This step calls OpenAI's API 1 time.\n\nAs illustrated, the primary cost is in the initial description generation (i.e., discussion by multi-persona collaboration). The generation is one-time and is completed **offline**, and would not delay the inference stage. Since NOT all triples are iterated, the cost remains acceptable even when the number of object/predicate categories increases. We respectfully refer the reviewer to {reviewer RK4c Q2} for the cost of online deployment.\n\n**Q3**: **Usage of possible coexistence descriptions.**\n\n**A3**: The goal of making the prediction of possible coexistence descriptions close to those of CLIP is to regularize the training process and avoid overfitting (L255-257). Such an approach brings more supervision signals during training and is inspired by the knowledge distillation-based open-vocabulary frameworks [b]. It is important to note that possible coexistence descriptions are used only during training. **During inference, they DO NOT contribute to the similarity measurement**, as the correlation $C_{r}^{n}$ equals 0 (see Eq. 1 & 2).\n\n**Q4**: **More ablative experiments on possible coexistence descriptions.**\n\n**A4**: In Table 8, $\\alpha_{indef}$ is a scaling factor (Eq. 8, L258) for training targets rather than a weight of a loss. Per your request, we remove $\\mathcal{L}_{indef}$ in Eq. 9 to study its influence. The results are as follows:\n\n| Method | Base |  | Novel |  |\n| --- | --- | --- | --- | --- |\n|  | mR @50 | mR @100 | mR @50 | mR @100 |\n| Ours | 12.3 | 14.7 | 25.2 | 31.5 |\n| w/o $\\mathcal{L}_{indef}$ | 12.1 | 14.3 | 23.6 | 30.1 |\n\nAs seen, the $\\mathcal{L}_{indef}$ leads to considerable improvements in the novel set (e.g., 1.6% mR@50 and 1.4% mR@100) and does not affect the performance of the base set. This further varifies our hypothesis (*i.e.*, regularize the training process and avoid overfitting as stated in L255-257). We will incorporate the above results into the appendix of our revision. Thanks.\n\n**Q5**: **Discussion on descriptions and CLIP's recognition capability.**\n\n**A5**: The generated description can be viewed as a **semantically informative** representation compared to the raw representation with only category names. They can convey extra concepts *without modifying model architecture or introducing additional learning processes*. Concretely, they 1) provide comprehensive concepts that help CLIP differentiate between relations (see Fig. 1 in [c]), and 2) offer simpler and more common concepts for relation categories that may not have been encountered during CLIP's pre-training (see Fig. 5 in [d]).\n\nNote that these detailed, fine-grained descriptions **may not align perfectly with CLIP's pre-training pipeline**. By collecting 10 horse images from the Web and comparing the vision-language similarities, we empirically find that CLIP struggles to properly distinguish between concepts with slight differences (*e.g.*, \"with four legs\" *vs*. \"with two legs\"). This indicates an inherent drawback of previous description-based methods that uniformly process all descriptions as affirmative classifiers (L50), and motivates us to introduce the renormalization mechanism (L69-76) so as to adaptively weight the generated descriptions. We will incorporate the discussions into the appendix of our revision. Thank you.\n\nReferences:\n\n[a] Unbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network, AAAI 2023.\n\n[b] Towards Open Vocabulary Learning: A Survey, TPAMI 2024.\n\n[c] Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models, NeurIPS 2023.\n\n[d] Visual Classification via Description from Large Language Models, ICLR 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
To all reviewers:

Thank you so much for your careful review and suggestive comments. We have revised our paper according to your comments. The major changes are as follows:

1. We improve the presentation of Sec. 3.1, according to Reviewer qa1m's comments.
2. We add an experiment to evaluate the performance of our model under standard settings, according to Reviewer qa1m's suggestion.
3. We add an experiment to evaluate the robustness of our model *w.r.t.* different base/novel splits, according to Reviewer qa1m's suggestion.
4. We clarify the motivation and working mechanism of our multi-persona collaboration, according to Reviewer qa1m's suggestion.
5. We add an experiment to evaluate the performance when using only one persona, according to Reviewer qa1m's suggestion.
6. We offer more detailed discussions regarding the generation process of descriptions, according to Reviewer qa1m's and xMLQ's comments.
7. We give a detailed discussion regarding the computational complexity, according to Reviewer RK4c's comments.
8. We clarify the usage of possible coexistence descriptions, according to Reviewer xMLQ's comments.
9. We add an ablation study on the effect of integrating possible coexistence descriptions into training, according to Reviewer xMLQ's comments.
10. We discuss the nature of descriptions and CLIP's recognition capability, according to Reviewer xMLQ's comments.

In addition, we include supplementary figures in the PDF of this "global" response for the following aspects:

1. The revised version of Fig. 1c.

Please refer to our response for more details. We have strived to address each of your concerns and welcome further discussions and insights.

Sincerely yours,

Authors.

Meta-review: The reviewers reached a positive consensus early on, and the authors successfully upheld this by effectively addressing their concerns. In their rebuttal, the authors presented several additional experimental results as requested by the reviewers, many of which further strengthen the paper. The AC also strongly recommends including these results in the camera-ready version.
Decision: Accept (poster)