Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models
Authors: Lin Li
Jun Xiao
Guikun Chen
Jian Shao
Yueting Zhuang
Long Chen
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual relation detection
Zero-short learning
Scene graph generation

Abstract: Pretrained vision-language models, such as CLIP, have demonstrated strong generalization capabilities, making them promising tools in the realm of zero-shot visual recognition. Visual relation detection (VRD) is a typical task that identifies relationship (or interaction) types between object pairs within an image. However, naively utilizing CLIP with prevalent class-based prompts for zero-shot VRD has several weaknesses, e.g., it struggles to distinguish between different fine-grained relation types and it neglects essential spatial information of two objects. To this end, we propose a novel method for zero-shot VRD: RECODE, which solves RElation detection via COmposite DEscription prompts. Specifically, RECODE first decomposes each predicate category into subject, object, and spatial components. Then, it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Different visual cues enhance the discriminability of similar relation categories from different perspectives, which significantly boosts performance in VRD. To dynamically fuse different cues, we further introduce a chain-of-thought method that prompts LLMs to generate reasonable weights for different visual cues. Extensive experiments on four VRD benchmarks have demonstrated the effectiveness and interpretability of RECODE.

Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models
Lin Li Jun Xiao', Guikun Chen', Jian Shao', Yueting Zhuang's Long Chen
'Zhejiang University "The Hong Kong University of Science and Technology
{mukti, junx, guikun. chen, jshao, yzhuang))0zju edu. en, longchen@ust..
(A) camyyig
IBI carying
having object in their hands persan having object their Carrying a person supporting an object hands hands Which of the pictures "nolding" and "carrying"?
Holdings
G holding
carrying
a person supporting object in their hands
AC are "nolding", and BD are "carrying
(b)
subject hands
subject hands subject is standing
subject is standing
holiing
(D) holding
holding
"O object with a handle a grip that held by the subject's hand
holding
carrying
subject with hands, with ams subject
sub is standing
subject with both hands
engaged position, perhaps walking or running subject with both hands and arms supporting heavy object
and arms supporting
subjest
heavy object
carrying
oog
subject more
engaged position,
CD tholding", and AB "carying"
peñhaps walking
or running
(c)
(d)
Figure 1: Illustration of the challenges of VRD with similar relation categories holding and carrying. Four images and their ground-trune are on the left. The subject and object for each triplct are denoted by blue and pink boxes, respectively. (a) A child may incorrectly identify these two relations only based on similar concepts alone. (b) Using class-based prompts, CLIP always maps these two relations adjacent locations the semantic space. (c) We humans always utilize composite visual cues to correctly distinguish between different relations. (d) Our proposed RECODE uses LLM (e.g., GPT) to generate composite descriptions that aid the CLIP model in distinguishing between them.
Abstract
Pretrained vision-language models, such as CLIP, have demonstrated strong gen- eralization capabilities, making them promising tools in the realm of zero-shot visual recognition. Visual relation detection (VRD) is typical task that identifies relationship (or interaction) types between object pairs within an image. However, naively utilizing CLIP with prevalent class-based prompts for zero-sho shot VRD has several weaknesses, e.g., it struggles to distinguish between different fine-grained relation types and it neglects essential spatial information of two objects. To this end, we propose novel method for zero-shot VRD: RECODE, which solves RE- lation detection via COmposite DEscription prompts. Specifically, RECODE first decomposes each predicate category into subject, object, and spatial components. Then, it leverages large language models (LLMs) to generate description- prompts (or visual cues) for each component. Different visual cues enhance the discriminability of similar relation categories from different perspectives, which significantly boosts performance in VRD. To dynamically fuse different cues, we further introduce a chain-of-then of-uhhuggtttttttth method that prompts LLMs to generate rea-
'Long Chen the corresponding author. Work was done when Lin Li visited HKUST.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

CLIP
Why X
with legs
hands
22.4
mounted or
15.9%
O with tool or equipment for mourting
painted
walking or
carrying
riding
oome
1904
mounted
object
object with mid distance
with back or surface for mounting
hortzontall sunyect above hodizontal
legs hooves
22.6
25.2
Spatial
CLIP with Visual
with legs
Why
with hands halding rains or other equipment seated on the back of the animal
W11
mounted 31.55
riding
Subject
O
21
with body
man-riding-sllphand
watzhing
riding
with four legs with saddle similar riding apparatus its back
carryin
60%
object
standing or 00%
vertical or horizontal ebove horizental object
sparta
with a mid to large distance
CLIP
Why X
with hand
lying on
holding
25.4
1277
Subject
arms
2442
covered
1695
holdingC
with handle or grip that held the human's hand +
looking
10.4%
object
vertical human baside under the square object with mid distance
laying
spoda
CLIP with Visuel Cues
Why
with arms and stretched out
laying
with limbs extanded
lying or
subject
with a fiat surface
child-laying on-bed
looking at sitting
covering
62%
12%
laying
object
with no movernntt
square horizontal subject horizontal
355
spoti
abjectwith or large distance
Figure 2: A comparative analysis of predictions made by RECODE and baseline CLIP using class- based prompts. illustrates how our method offers interpretability to the relation classification results through the similarity & between the image and the description-baser prompts.
sonable weights for different visual cues. Extensive experiments on four VRD benchmarks have demonstrated the effectiveness and interpretability of RECODE.
Introduction
Recent advances in pretrained vision-language models (VLMs) [1, 2, 3, 4] (e.g., CLIP [T]) have shown remarkable generalization ability and achieved impressive performance zero-shot recogni- tion tasks. Specifically, CLIP employs two encoders: an image encoder that converts images into visual features, and a text encoder that transforms sentences into semantic features. This design allows the encoders to map different modalities into a common semantic space. When the inputs to the text encoder are class-based prompts, such as "A [CLASS]", "A photo of [CLASS]', CLIP can compare the image and prompts in the shared semantic space, thereby enabling zero-shot recognition of novel categories [1]. Compared to object recognition, visual relation detection (VRD) is much more challenging, which needs to identify the relation types between object pairs within an image in the form of (subject, relation on, object) [5, 6, 7, 8, 9]. differs from object recognition in that requires an understanding of how objects are related to each other. By crafting class-based prompts to describe these relation types, CLIP could potentially be extended perform zero-shot VRD.
However, this straightforward baseline presents notable challenges. Imagine you are a child asked to distinguish relation categories "holding" and "carrying", both involving person and an object. Based on the similar concepts of "holding" (i.e., a person having an object in their hands) and "carrying" (i.e., a person supporting an object in their hands), it would be difficult to determine the correct prediction (cf., Figure (aa).. In other words, class-based prompts of "holding" and "carrying" might be projected to adjacent locations in semantic space by CLIP, leading a relation sensitivity issue: CLIP struggles to differentiate between the subtle nuances of similar relations. Secondly, class-based prompts overlook the unique spatial cues inherent to each relation category, leading spatial discriminability issue. The "holding" category generally suggests the object being at certain height and orientation relative to the person, while "carrying" implies a different spatial position, typically with the object located lower and possibly supported by the person's 's entire body. The neglect of spatial cues leads to inaccuracies distinguishing between such apatial-aware relation categories. Moreover, applying CLIP in this manner brings about a computational efficiency issue. Using CLIP requires cropping each union region of a subject-object pair separately from the original image (i.e., 22 crops for N proposals), leading to computational inefficiencies.

---

Nonetheless, we humans can distinguish relation categories from different visual cues. For example, from the subject's 's perspective, we could think that in the case of "holding", a person might be standing while having an object, such as umbrella, in their hand. Meanwhile, in the case of "carrying", a person should be in a more engaged position, perhaps walking or running with both hands and arms supporting a heavy object, like a suitcase. In addition, spatial cues also play an important role in identifying these relation categories. For example, when a person is carrying an umbrella, the umbrella is usually positioned lower and closer to the person's 's body compared to when the person is holding an umbrella. Based on these visual cues, we can easily identify scenarios such as "person-hool holding-umbrr
and cearson-caryingg-i
"arrelll"" la" in Figure I(c).
Inspired by our humans' ability to extract and utilize different visual cues, we present a novel method for zero-shot VRD: RECODE, which classifies RElation via COmposite DEscriptions. It first uses large language models (L..Ms) [10], to generate detailed and informative descriptions for different components of relation categories, such as subject, object, and spatial. These descriptions are then used as description- based prompts for the CLIPP model, enabling it to focus on specific visual features that help distinguish between similar relation categories and improve VRD performance. Specifically, for the subject and object components, these prompts include visual cues such as appearance (e.g.. with leg), size (e.g., small), and posture (e.g., in sitting posture). For the spatial component, these prompts include cues related to the spatial relationships between objects, such as relative position and distance. By incorporating different visual cues, RECODE enhances the discriminability of similar relation categories, such as "riding" and "mounted" based on the different postures of the subject, e.g., "seated the back of animal" for the subject of "riding". Similarly, spatial visual cues can be used to differentiate between "laying on" and "holding" based on the relative position between the subject and object, such "subject above object" and "subject under object" (ef., Figure 2).
In addition, we explore the limitations of several description generation prompts for visual cue, e.g., relation class description prompt [11], and then design guided relation component description prompt that utilizes the high-level object categories to generate more accurate visual cues for each relation category. For instance, if the high-level category of object is "animal"" the generated object descriptions for relation "riding" are tailored to the "animal" category, e.g., "with four legs", instead of the "product", e.g., "with wheels". Mcanwhile, to better fuse the evidence from different visual cues, we further leverage LLMs to predict reasonable weights for different components. Particularly, we design chain- -f-thuughtt (CoT) method [12] break down this weight assignment problem into smaller, more manageable pieces, and prompt L.M to generate a series of rationales and weights.
To evaluate our RECODE, we conducted experiments four benchmark datasets: Visual Genome (VG) [13] and GQA [14] datasets for scene graph generation (SGG), and HICO-DET [15] and V-COCO [16] datasets for human-object interaction (HOI) detection. Experimental results prove the generalization and interpretability of our method. In summary, we made three main contributions in this paper: 1) We analyze the weaknesses of the prevalent class- based prompt for zero-shot VRD in detail and propose a novel solution RECODE. RECODE leverages the power of LLMs to generate description bassed prompts (visual cues) for each component of the relation class, enhancing the CLIP model's 's ability to distinguish between various relation categories. 2) We introduce a chain-of-thought method that breaks down the problem into smaller, more manageable pieces, allowing the LLM to generate a series of rationales for each cue, ultimately leading to reasonable weights for each component. 3) We conduct experiments on four benchmark datasets and demonstrate the effectiveness and interpretability of our method.
2 Approach
Typically, VRD is comprised of two sub-tasks: object detection and relation classification [5]. Since zero-shot object detection has been extensively studied [17, 1, 11], in this paper, we primarily focus on zero-shot relation classification. Specifically, given the bounding boxes (bboxes) (b,) and object categories (o:) of all objects, our target is to predict the visual relation (or predicate/interaction) categories } between pairwise objects. To facilitate presentation, we use 8, o, and p to denote the subject, object, and their spatial position in a triplet respectively, and to denote the relation category.
Class-based Prompt Baseline for Zero-Shot VRD. Following recent zero-shot object recognition methods, straightforward solution for zero-shot VRD is the CLIP with class-based prompt. Specif-
We description to represent visual cue.
3

---

Visual Feature Dacomposing
Semantic Feature Decomposing
Relation Classification
Text Encoder
Ari nillng
St
Me
subject
Image Encoder
A
Ma
$40
Ama
Mommomen, with legs Momon, hands
woman-horse
Encoder
Woman seated the back animal
Image Encoder
Score
talelas
du, horse, body
sitting on
Decomposing
Encoder
dusi horse, with four legs saddle similar nicinng apparatus back
laying on
riding
looking at
Encodar
Text Encodar
4px- vertica red bux above horizontal green box with mid cistance
nying
Figure 3: The framework of RECODE. 1) Visual feature decomposing decomposes the triplet into subject, object, and spatial features. 2) Semantic feature decomposing decomposes relation categories into subject, object, and spatial descriptions. 3) Relation classification calculates similarities between decomposed visual and semantic features and applies softmax to obtain the probability distribution.
ically, a pretrained CLIP image encoder (-) and a pretrained CLIP text encoder T() are used to classify pairwise objects with set of relation classes. For each relation class, a natural language class-based prompt pa is generated, incorporating the relation information, e-g. "[REL-CC ECLL- -J]-ing/ed" or "a photo of [REL-CLS]" Each prompt is then passed through T(-) get semantic embedding t, while the union region of a subject-object pair is passed through (-) to get visual embedding v. The cosine similarity between v and & of different relation categories is calculated and processed by softmax function to obtain the probability distribution over all relation categories.
2.1 Zero-shot VRD with Composed Visual Cues
To overcome the limitations of class-based prompts, we propose novel approach RECODE for zero-shot VRD. It consists of three parts: visual feature decomposing, semantic feature decomposing, and relation classification (cf., Figure 3). In the first two parts, we decompose the visual features of the triplet into subject, object, and spatial features, and then generate semantic features for each component. In the last part, we calculate the similarities between the decomposed visual features and a set of semantic features, and aggregate them to get the final predictions over all relations.
Visual Feature Decomposing. To enhance spatial discriminability and computational efficiency, we decompose the visual features of a triplet into subject, object, and spatial features. For subject and features, we crop the regions of the subject and object from the original image using the given bboxes b, and ba and encode them into visual embeddings v. and v, using the image encoder (-)) of CLIP. For spatial features, aim to obtain the spatial relationship between the subject and object based on their bounding boxes. However, directly obtaining all spatial images based on the given bounding boxes is computationally expensive due to the diversity of spatial positions (Nv2 each image). To address this, we simulate the spatial relationship between the subject and object using finite set of spatial images, represented by red and green bboxes respectively. We define four attributes (shape, size, relative position, and distance) based on bounding box properties. Each attribute is assigned finite set of values to construct finite set of simulated spatial images. For given triplet, we match the calculated attribute values with the most similar simulated image". The matched spatial image is then encoded into visual embedding U, using V--) of CLIP.
Semantic Feature Decomposing. To improve the CLIP model's 's ability to distinguish between prompts D to augment the
different relation classes, incorporate set of description- original class- based prompt for each relation category. For the subject and object components, we generate set of description-basee prompts D. and D. to provide additional visual cue information, the generation process is described in Sec. 2.2. These prompts contain object categories with specific visual cues that highlight the unique characteristics of the relation being performed, e.g., "women, with legs", which enhances the discriminability between similar relation categories. For the spatial component, only contains set of description-based prompts D, that include information about the relative position and distance between the subject and object in the image. By incorporating
Due to the limited space, the details are left in the appendix.

---

(a) Relation Class Description
(b) Relation Companent Descripton
(c)
Guided Relation Component D Description
What are useful features for disttingiihhnng a relation "riding" photo?
What are the useful visual features subject object can distinguish the relation "riding" a photo?
What are the useful visual features of subject and object that can distinguish the relation "riding'. when subject belongs to 'human" and object belongs to "animal" ?
The direction movement and speed of the person and the mode of transportation. surroundings, such trail arena, that recreational competitiee riding context.
The subject may be holding onto reins handles attached to the object
The subject may be holding reins or some sort of device used for directing the animal's movement. The object may have hamess saddle on its body, which is often used for riding or controlling the animal.
The object may have saddle or other equipment spc ifically
designed for riding.
Figure 4: Examples of different prompts used for generating descriptions of visual cues. (a) Relation class description generates descriptions for each relation class directly. (b) Relation component description generates descriptions for each component of the relation separately. (c) Guided relation component description incorporates high-level object category guide generation process.
this additional information, we aim to distinguish between relations based on spatial location. After generating these sets of description-baat and {td,,) using& text encoder T(-), separately. These embeddings, along with the class-based prompt embedding for are used for relation classification.
prompts, we obtain semantic embeddings {fd,, J. {tdo
Relation Classification. In this step, we compute the similarity score between the visual and semantic features to obtain the relation probability distribution. We first calculate the cosine similarity ((::)) between each visual embedding and semantic embedding for each relation category The final score incorporates both class-based and description-biie prompts, and is calculated as follows:
Srr) = <D((vs)e)) (o)o
Wkk [D(
Σ Mex Lake
(I)
kE(..o.p)
dA, (ppir)
classi bused promits
description-hased prompes
where Wi represents the importance of visual cues for each component E {s, o, and [r) denotes the number of visual cues in Dz(r) for relation category We compute the similarity of individual visual cues for each component and then obtain their average. The weights of different components determined by LLM, which will be discussed in Sec. 2.2. Finally, apply softmax operation to the scores to obtain the probability distribution over all relation categories.
2.2 Visual Cue Descriptions and Weights Generation
LLMss, such as GPT [10), have been shown to contain significant world knowledge. In this section, we present the process of generating deseriptions of visual D,, Da, and Dp. well as the weights Ws, wo, and Wp for each component of each relation category using L.MMs.
2.2.1 Visual Cue Descriptions
In this section, we explore methods for generating descriptions of visual cues for relation decompo- sition. Inspired by the work [11] of zero-shot image classification, we first propose relation class description prompt, which generates descriptions from the perspective of class-level (cf., Figure 4(a)). has the advantage of producing descriptions that are easy to interpret and understand. However, it may result in overly diverse and information-ride descriptions that could hinder the extraction of meaningful visual cues, e.g., "speed of the person" in Figure 4(a).
To address this limitation, we then consider another relation component description prompt, which involves decomposing the relation into its subject and object components and generating descriptions of their visual features separately (cf., Figure 4(b)). While this type of prompt allows for more focused and specific descriptions of visual cues, may not be effective in capturing the variations visual features between different subject-object category pairs. For example, shan-riding-horss and "person-riding bike" typically have totally different visual features for the object. The visual cues "reins" and "saddle" of the object in Figure 4(b) are inappropriate for a "bike".
5

---

(8) Weight P Prompt
(b) Weight P Prompt with Chain-of-Thoughe
Suppose you are relation classification model.
The sum of weights must be 1.0!
Given: subject belongs to "human" and object belongs "product".
A: Let's think step by step! (CoT prompt)
The visual features of subject with eyes directed towards the object, with head upright.
First, we need to consider the importance of the subject's visual features: Since the direction of the eyes and head position strongly indicate the focus attention, we will give them a weight of 0.6.
The visual features of object with ible features such as front, display, or screen.
The visual features of position: subject positioned either above, below, left or right of the object at mid distance. Q: How do you weight these visual features to determine the relation is "looking at"? The sum of weights must be 1.0!
Next, we need to consider the importance of the object's visual features. Since the visible features such as front, display, or screen indicate that the object is something that can be looked Finally.... we will give them weight of 0.1.
will give them weight 0.3.
Therefore, we can weight these visual features as follows: > Visual features of subject: 0.6
Visual features subject 0.4
uas features of object: 0.4
Visual features of object: 0.3
Visual features posibion: 0.2
Visual features position: 0.1
Figure 5: Illustration of the effectiveness of CoT method in generating reasonable visual cue weights. (a) Prompt without COT. LLM assigns same weights for subject and object. (b) Prompt with CoT. LLM analyzes the importance of each cue step by step and assigns more reasonable weights.
Therefore, we design guided relation component description prompt. It builds upon the second method by incorporating the high-level category information of the object into the generation process, leading to more accurate and informative descriptions of the visual features of both the subject and object components (cf., Figure 4(c)). To achieve this, we classify the object into high-level classes, such as "human", "animal", and "product", to guide the description generation. For example, "bike" is classified as "product", and "horse" is classified "animal", This allows for the separate generation of visual feature descriptions for each high-level object class, e.g., "a harness or saddle on its body" for "animal", resulting more precise and relevant visual cues for each relation category'.
2.2.2 Visual Cue Weights
Intuitively, different combinations of visual cues may have varying degrees of importance in relation classification. For example, for relation "looking at", the visual cue "with visible features" of the object may not be as informative as the visual cue "with eye" of the subject. To account for this, we leverage the impressive knowledge and reasoning abilities of LLMs to analyze the discriminative power different visual cues and dynamically assign weights accordingly. Specifically, we provide each combination of visual cues as input to LLM and prompt it to determine the appropriate weight for each cue for distinguishing the given predicate. The prompts used for this purpose are Figure 5.
Chain-of- of-TToughtttt (CoT) Prompting. To ensure the generated weights are reasonable, we utilize a CoT method that has demonstrated remarkable reasoning abilities [12, 18]. Specifically, we prompt the LLM to generate rationales by using the stepwise reasoning prompt "Let's think step by step!" to break down the problem into smaller, more manageable pieces. Then LLM generates a series of rationales, and those that lead to the reasonable weights. For example in Figure 5, we demonstrate the importance of the CoT method in generating more accurate weights. Without the stepwise reasoning prompt, LLM generates the same weight for both the subject and object visual cues for "looking at", which clearly unreasonable. However, with the CoT prompt, LLM able to analyze each cue step by step, leading to more accurate assignment of weights, i.e., the cues about the subject are relatively more important. In order to standardize the format of the strings generated by LLMs for extracting different components of visual cues and weights, we make certain modifications to the prompts for descriptions and weights
3 Experiment
3.1 Experiment setup
Datasets. We evaluated our method on four zero-shot VRD benchmarks: 1) VG [13] contains 26,443 images for testing, each annotated with object and predicate labels to form a scene graph. Following previous works [5], we used the pre-processed VG with 150 object classes. We adopted the 24 semantic predicate classes proposed in [19, 20], as they are more informative and challenging for classifying. 2) GQA [14] is a large-scale SGG dataset. We used the same split provided by [21],
which contains 8, 8,208 images for testing with 200 object classes. As for predicate classes, selected

---

Table Evaluation results on the test set VG and GQA datasets. denotes removing the guidance from high-level object category. * denotes integrated with Filter strategy.
Predicate Classification
Data
Method
R@20
A
R@50
A
R 100
mR mR
A
mR@50
mR 100
A
CLS
7.2
10.9
13,2
9.4
14.0
17.6
CLSDE
7.0
-0.2
10.6
-0.3
12.9
-0.3
8.5
-0.9
13.6
-0.4
16.9
-0.7
VG
RECODE
7.3
0.1
11.2
0.3
15.4
2.2
8.2
-1.2
13.5
-0.5
18.3
0.7
RECODE
9.7
2.5
14.9
4.0
19.3
6.1
10.2
0.8
16.4
2.4
22.7
5.1
RECODE
10.6
3.4
18.3
7.4
25.0
11.8
10.7
1.3
18.7
27.8
10.2
CLS
5.6
7.7
9.9
6.3
9.5
12.2
CLSDE
5,4
-0.2
7.2
-0.5
9.3
-0.6
6.0
-0.3
8.8
-0.7
11.5
-0.7
GQA
RECODE
5.2
-0.4
7.8
0.1
10.2
0.3
5.8
-0.5
8.9
-0.6
11.3
-0.9
RECODE
6.3
0.7
9.4
1.7
11.8
1.9
7.8
1.5
11.9
2.4
15.1
2.9
RECODE *
7.0
1.4
11.1
3.4
15.4
5.5
9.4
3.1
14.8
5.3
20.4
8.2
26 semantic predicate classes by referring to VG. 3) HICO-DET [15] contains 9,658 testing images annotated with 600 HOI triplets derived from combinations of 117 verb classes and 80 object classes. 4) V-COCO [16] comprises 4,946 testing images annotated with 29 action categories.
Evaluation Metrics. For SGG datasets (i.e., VG and GQA), we reported Recall K (R@K) which indicates the proportion of ground-truths that appear among the top-K confident predictions, and mean Recall K (mR@r @) which averages R@K scores calculated for each category separately [22].
For HOI datasets (i.e., HOI-DET and V-COCO), reported mean Average Precision (mAP) [23].
Implementation Details. For the LLM, we employed the GPT-3.5-turbo highly performant variant of the GPT model. As for CLIP, we leveraged the OpenAl's publicly accessible resources, specifically opting for the Vision Transformer with a base configuration (ViT-B/32) as default backbone
Settings. The bounding box and category of objects were given in all experiments. We compared our RECODE with two baselines: CLS, which uses relation-CLasS to compute the similarity between the image and text. 2) CLSDE, which uses prompts of relation CLasS DEscription as shown in Figure 4(a). Each component of the proposed framework can serve as plug-and-play module for zero-shot VRD. Specifically, 1) Filter, which denotes filtering those unreasonable predictions (e.g., aud-eating-house
prompts (e.g.. "riding")
with the rules generated by GPT3, 2) Cue, which denotes using description-based prompts (Sec. 2. 3) Spatial, which denotes using spacial images as additional features. 4) Weight, which denotes using dynamic weights generated by GPT to determine the importance of each feature, i.e., visual cue weights.
3.2 Results and Analysis
In this work, we evaluated the prediction performance of the proposed framework on two related tasks, i.e., SGG and HOI. The former outputs a list of relation triplet prud, obj), while the latter fix the category of sub to human. Overall, our method achieved significant improvement on the two tasks compared to the CLS baseline, which shows the superiority of our method.
Evaluation SGG. From the results in Table I, we have the following observations: 1) CLSDE showed worse performance than the trivial CLS baseline. This because the considerable noise in CLSDE which may hinder the model to attend the most distinguishable parts. 2) With the proper guid- ance, RECODE achieved considerable improve- ments compared to the baselines, e.g., 0.8% to 6.1% gains on VG and 0.7% 2.9% gains on GQA. The performance drops of RECODET also demonstrated the importance of guidance from high-level ob- ject categories during the generation process. 3) Integrated with the filtering strategy, i.e., RECODE*, achieved the best performance over all metrics, which suggests that commonsense knowledge is com- plementary and effective for zero-shot VRD. also demonstrated that CLIP struggles to distinguish abstract concepts, i.e., relation sensitivity mentioned in Sec. 1.
Table 2: Evaluation results on the test set of HICO-DET and V-COCO datasets.
HICO-DET
V-COCO
Method
Full Rare Non-Rare Rare
Role
Role 2
CLS
32.3 33.2
31.8
25.5
28.6
CLSDE
32.5 33.1
32.2
25.6
28.8
RECODE
32.5 33.0
32.4
25.7
28.8
RECODE
32.7 33.2
32.5
26.0
29.0
7

---

Table 3: Ablation studies on different architectures CLIP. The official released weights are used.
Predicate
Classification
Architecture
Method
R@
R@ 50
R@ 100
mR@20
mR @5
mR@ 100
CLS
8.3
15.0
21.5
7.6
14.2
24.2
ViT-L/14
RECODE
11.2
19.9
28.0
9.1
18.5
28.1
CLS
8.6
13.4
21.8
7.7
13.9
23.0
ViT-L/14@336px
RECODE
12.1
21.1
29.2
9.7
19.5
28.2
CLS
7.5
13.7
19.4
9.1
13.9
24.0
ViT-B/32
RECODE
10.6
18.3
25.0
10.7
18.7
27.8
CLS*
8.6
15.5
22.1
9.8
17.2
25.2
ViT-B/16
RECODE
12.6
21.0
28.5
12.5
20.2
30.0
Evaluation HOL Since standard evaluation procedure of HOI had filtered those unreasonable predictions, RECODE was not evaluated here. From the results in Table 2, we can observe that the performance gains were lower than those on SGG, e.g., 0.0% 0. 0.7% gains on HICO-DET and 0.4% to 0.5% gains on V-COCO. The reasons are two-fold. On the one hand, since the category of subject is always human, its features some of the actions are very similar in appearance. For example, distinguishing between actions like "person throw-ssoor -sporsss ball" and "person- carron--catc--e spootsss ball" is challenging due their visual similarity.
similar be distinguished by CLIP. On the other hand,
3.3 Diagnostic Experiment
Architectures. We investigated the impact of changing the architectures of CLIP, as shown in Table 3. From the results, we can observe consistent improvements regardless of the architecture used.
Key Component Analysis. The results summarized in Ta- ble 4. The first row refers to the CLS baseline. Four crucial clusions can be drawn. First, with the guidance of Cue, sistent improvements can be ob- served, e-g., 0.2% to 3.4% gains on R@K w/o Filter and 1.3% to 4.1% gains on @@K with Filter. Second, by introducing the spa- tial feature, the relative position of subject and object is consid- ered, resulting notable perfor-
Table 4: Analysis of key components on the test set of VG.
Predicate Classification
Filter Cue Spatial Weight R 20 R 50
R@ 100 mR @:50 @ 100
7.2
10.9
13.2
9.4
14,0
17.6
7.4
12.3
16.6
9.0
14.0
19.5
9.1
17.4
9.3
15.0
20.3
7.9
13.4
17.7
9.3
14.7
20.5
9.7
14.9
19.3
10.2
16.4
22.7
7.5
13.7
19.4
9.1
15.9
24.0
8.8
15.9
23.5
10.3
17.2
26.2
9.3
16.3
22.5
10.1
18.1
25.5
10.0
24.8
10.4
17.8
26.7
10.6
18.3
25.0
10.7
18.7
27.8
mance gains on @@ K (0.8% to 1.7%) and mR@K (0.3% to 1.0%) w/o Filter compared to just using Cue. This is because the spatial feature is of importance for relation detection [6]. Third, benefiting from the impressive reasoning ability of LLMs, the proposed weighting strategy determine the importance of different cues, thus achieving further improvements, e.g., 0.5% 1.1% gains on compared average aggregation. Fourth, by filtering those unreasonable predictions, consistent improvements can be observed. The reason may be that the performance relation detection of CLIP is not accurate enough. Empirically, commonsense knowledge is a feasible way to filter those noise. Combining all components allows for the best overall performance on evaluation metrics.
Case study. To investigate the most important regions for distinguishing relations, we visualized the attention map given different images and prompts (cf., Figure 6). From the visualization of class-based prompts, we can observe that CLIP may attend those regions unrelated to the query prompts, e.g., focusing on the body of person given relation "growing on". We attribute this phenomenon to insufficient information within given prompts, which is also our motivation to introduce visual cue descriptions. As for description-brr based prompts, CLIP can attend to right regions with the guidance of descriptions, e.g., focusing colorful patterns the product given relation "painted on".

---

Class-based Prompts
Description-based Prompts
Subject Descripton: animal-product
eating: girafle, with head or mouth toward hanging from: girafle, with limbs
giraffe-eabin-
eating yh: 26.1
hanging from di 27.2
eating dr: 33.1
hanging from 30:30
Subject Description: human-product
using: person, poss Ibly with fingers actively interacting with something
looking at person, with eyes directed towards something
person-using phone
using 4:23.4
looking at d: 25.2
using d: 25.2
looking at 0:
Object Description: product<rodduct
growing on: tree, larger, stationary with sturdl
structure that accommodate growing product hanging from: tree, with hook, handle
eal-ggrowing on-iree
growing 23 hanging from 24.5
growing 27.7.3 hanging from 23.4
Object Descriptio:: product-produc
painted vase, the painted design or image may cover part of the its body mounted on: attachment mechanism
flower-painted painted d: mounted 26.44 Figure 6: Visualization of CLIP attention maps input images with different prompts. The right side shows the partial deseription-prompe generated for each predicate category given the high-level object category. They used to generate the corresponding attention maps the right.
painted 9:32.3
d:29.5
4
Related Work
Visual Relation Detection (VRD) aims to predict the relationships of given subject-object pairs, which be viewed pair-wise classification task and have been widely studied in the image domain, e-g., scene graph generation (SGG) [5, 6, 22, 24] and human-object interaction (HOI) detection [25, 26, 27]. Previous solutions mainly focus on learning representations from the training samples on pre-defined categories, which may suffer from noisy annotations [22] or long-tailed predicate distribution [6, 28] and far from the needs of the real-world scenarios. Recently, some attempts [8, 29] adopted prompt-tuning [30] to predict unseen categories during inference. However, since the learnable prompts may be overfitting when trained on seen categories, their performance is sensitive to the split of seen/unseen categories [31]. In contrast, our method can predict the relationships directly without any training samples, and has better interpretability and generalization ability, especially in rare informative relation categories.
Zero-shot Visual Recognition enables the model recognize categories that it has during training, which is one of the research hotspots in the vision community. Aligning visual representations pre- pre-trained word embeddings (e.g., Word2Vee [32] and GloVe [33]) is intuitive and feasible way to achieve this goal [34]. More recently, VLMs, which use contrastive learning [35] learn a joint space for vision and language, have demonstrated their impressive zero-shot ability [1]. Therefore, many zero-shot works [36, 37, 38] adopted such VLMs as their basic component to use the knowledge of the learned joint space. However, most of them only utilized the class name of unseen categories during inference, which makes an over-strong assumption that the text encoder project proper embeddings with only category names [11]. Then, Menon and Vondrick [11] proposed to query L.LMs for the rich context of additional information. Nonetheless, it non-trivial to apply such paradigms to VRD as discussed in Sec. 1. To the best our knowledge, we are the first to leverage both LLMs and VLMs for VRD in an efficient, effective, and explainable way.

---

5 Conclusion
In this paper, we proposed a novel approach for zero-shot Visual Relationship Detection (VRD) that leverages large language models (LLMs) to generate detailed and informative descriptions of visual cues for each relation category. The proposed method addresses the limitations of traditional class-based prompts and enhances the discriminability of similar relation categories by incorporating specific visual cues. Moreover, we introduced chain-of-thought method that breaks down the problem into smaller, more manageable pieces, allowing the LLM to generate series of rationales for each visual cue and ultimately leading to reasonable weights. Our experiments on four benchmark datasets demonstrated the effectiveness and interpretability of our method.
Acknowledgement. This work was supported by the National Key Research Development Project of China (20212D01 the National Natural Science Foundation of China (U19B2043, 61976185), and the Fundamental Research Funds for the Central Universities (226-2023-00048) Long Chen is supported by HKUST Special Support for Young Faculty (F0927), and HKUST Sports Science and Technology Research Grant (SSTRG24EG04).
References
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In JCML, pages 8748- $748-8763, 2021.
[2] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, pages 7959-7971, 2022.
[3] Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Demoncourt, Trung Bui, and Mohit Bansal. Fine-grained image captioning with clip reward. In Findings of NAACL, 2022.
[4] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In /CML, pages 4904-4916, 2021.
[5] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, pages 5410-541 2017.
[6] Kaihua Tang. Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In CVPR, pages 37 3716-3725, 2020.
[7] Xingchen Li, Long Chen, Jian Shao, Shaoning Xiao, Songyang Zhang, and Jun Xiao. Rethinking the evaluation of unbiased scene graph generation. In BMVC, 2022.
[8] Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Towards open-vocabulary scene graph generation with prompt-based finetuning. In ECCV, 2022.
[9] Lin Li, Guikun Chen, Jun Xiao, Yi Yang, Chunping Wang, and Long Chen. Compositional feature augmentation for unbiased scene graph generation. In ICCV, pages 21685-21695, 2023.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NIPS, 2020.
[11] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2022.
[12] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NIPS, 2022.
[13] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
10

---

[14] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real- visual reasoning and compositional question answering. CVPR, pages 7700-6709, 2019.
[15] Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. Hico: A benchmark for recognizing human-object interactions in images. In ICCV, pages 1017-1025, 2015.
[16] Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint 1rXiv:1505.04474, 2015.
[17] Caixia Yan, Xiaojun Chang, Minnan Luo, Huan Liu, Xiaoqin Zhang, and Qinghua Zheng. Semantics-guided contrastive network for zero-shot object detection. TPAMI, 2022.
[18] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting large language models. arXiv preprint wxXi:2210.033993, 2022.
[19] Neural motifs: Scene graph parsing with global context. In CVPR, 2018.
[20] Anh Duc Bui, Soyeon Caren Han, and Josiah Poon. Sg-shuffle: Multi-aspect shuffle transformer for scene graph generation. In Australasian Joint Conference on Artificial Intelligence, pages 87-101. Springer, 2022.
[21] Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, and Liqiang Nie. Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In CVPR, pages 19427-19433 2022.
[22] Lin Li, Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang, and Jun Xiao. The devil is in the labels: Noisy label correction for robust scene graph generation. In CVPR, pages 2022.
[23] Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In WACV, pages 381-389, 2018.
[24] Yao Teng and Limin Wang. Structured sparse r-cnn for direct scene graph generation. In CVPR, pages 19437-19446, 2022.
[25] Keizo Kato, Yin Li. and Abhinav Gupta. Compositional learning for human object interaction. In ECCV, pages 234-251, 2018
[26] Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. Detecting human-object interactions with action co-occurrence priors. In ECCV, pages 718-736, 2020.
[27] Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In CVPR, pages 20123- 20132, 2022.
[28] Guikun Chen, Lin Li, Yawei Luo, and Jun Xiao. Addressing predicate overlap in scene graph generation with semantic granularity controller. In ICME, pages 78-83. IEBE, 2023.
[29] Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. Compositional prompt tuning with motion cues for open-vocabulary video relation detection. In ICLR, 2023.
[30] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys, 55(9):1-35, 2023.
[31] Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-box labels. In ECCV, pages 266-282, 2022.
[32] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3788 2013.
[33] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, pages 1532-1543, 2014.

---

[34] Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. CVPR, pages 6857-6866, 2018.
[35] Ching-Yan Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In NIPS, 2020.
[36] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, pages 14392-14402, 2021.
[37] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.
[38] Tao Wang and Nan Li. Learning detect and segment for open vocabulary object detection. In CVPR, 2023.
12


Reviews:
[{"review_id": "GZIJUXYC0K", "reviewer": "Reviewer_iRhq", "summary": "Naively utilizing CLIP with prevalent class-based prompts for zero-shot VRD has several weaknesses, e.g., it struggles to distinguish between fine-grained relation types and neglects essential spatial information of two objects. To this end, the authors propose a novel method for zero-shot VRD: RECODE, which solves RElation detection via COmposite DEscription prompts. Specifically, RECODE first decomposes each predicate category into subject, object, and spatial components. Then, it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Different visual cues enhance the discriminability of similar relation categories from different perspectives, boosting performance in VRD. To dynamically fuse different cues, they introduce a chain-of-thought method that prompts LLMs to generate reasonable weights for different visual cues.", "questions": "Please refer to the questions in the weakness.", "limitations": "I cannot find a potential negative societal impact in this paper.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The framework for decomposing visual cues and using LLM to separately generate prompts for subject, object, and spatial features seems novel.\n- The proposed method shows noticeable performance improvements, and the authors provided an ablation study to solidly analyze the design choices of the proposed method.", "weaknesses": "- The baselines in the experiments seem weak. Are the baseline methods recent enough models? To verify the effectiveness of the proposed method, the RECODE should be attached to the recent state-of-the-art model and show consistent performance improvement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We are willing to address all your questions.\n\n## Incorporated to More Recent SOTA Models\n**Table_R 1**: Performance of combining with different SOTA pre-trained visual-language models on VG dataset. CLS$^\\star$ denotes the model uses class-based prompts to compute the training-free zero-shot similarity between the image and text.\n\n| Backbone                   | Method             | R@20  | R@50  | R@100 | mR@20 | mR@50 | mR@100 |\n|--------------------------  |------------------  |------ |------ |-------|-------|-------|--------|\n| MS-CLIP [1]              | Baseline (CLS$^\\star$) | 8.2   | 15.1  | 21.5  | 7.9   | 16.4  | 22.4   |\n|                           | RECODE$^\\star$     | **9.2** | **17.3** | **24.7** | **8.3** | 15.4  | **22.6** |\n| DECLIP [2]              | Baseline (CLS$^\\star$) | 11.0  | 18.3  | 24.4  | 11.0  | 19.0  | 27.1   |\n|                           | RECODE$^\\star$     | **11.4** | **19.3** | **25.9** | 10.5  | **19.5** | **27.8** |\n\nWe acknowledge the importance of comparing our proposed approach with the recent state-of-the-art (SOTA) models in the field. Since our methods are training-free (cf, the **global response** for clarifying experimental settings), we combined our methods with other SOTA pre-trained visual-language (VL) models, e.g., MS-CLIP [1] and DECLIP [2], which can achieve training-free zero-shot VRD. The results are reported in Table\\_R 1. Notably, when combining RECODE$^\\star$ with these different SOTA VL models, we also observed considerable performance gains compared to the baseline CLS$^\\star$. These consistent improvements underline the effectiveness and generalizability of our RECODE.\n\n[1] Learning visual representation from modality-shared contrastive language-image pre- training. In ECCV, 2022. \\\n[2] Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In ICLR, 2022.\n"}, {"review_id": "Nzd0AShgjs", "reviewer": "Reviewer_CZ8p", "summary": "This paper presents RECODE, a novel method for zero-shot visual relation detection (VRD), designed to address the shortcomings of models like CLIP in distinguishing subtle relation categories and spatial discriminability. RECODE leverages large language models (LLMs) to generate detailed description-based prompts for each relation class component, thereby enhancing VRD performance. The authors also introduce a chain-of-thought method that breaks down the problem into smaller parts for LLMs, thereby assigning reasonable weights for each component. The effectiveness and interpretability of the method are demonstrated through experiments on four benchmark datasets.", "questions": "1. The paper mentions the incorporation of specific visual cues to improve the discriminability of related categories. Could you provide more details on how these cues were selected? What criteria were used to determine their relevance and effectiveness? Additionally, did you consider any alternative visual cues during the experimentation process? Exploring different visual cues and discussing their impact could provide further insights into the effectiveness of the proposed approach.\n\n2. The paper demonstrates improvements in relation classification for zero-shot VRD, but it would be valuable to discuss the scalability of the proposed approach. How does the performance scale with an increasing number of related categories and visual concepts? Are there any computational or efficiency limitations that arise when dealing with larger datasets or more complex scenes? \n\n3. The paper should provide a clear justification for the choice of evaluation metrics used to assess the performance of the proposed approach. Are there any limitations or biases associated with the selected metrics? Additionally, it would be helpful to include a discussion on the limitations of these metrics in capturing the true performance of zero-shot VRD models.\n\n4. While the paper discusses the experimental results and improvements over baseline methods, it would be valuable to have a section dedicated to the limitations of the proposed approach. Identifying and addressing these limitations can help guide future research directions. Additionally, it would be beneficial to have a discussion on potential extensions or improvements to the proposed approach that could further enhance its performance or broaden its applicability.", "limitations": "1. Auhors has not included the section or discussion on limitations and, if applicable, the potential negative social impact of their work. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The approach introduces a novel framework, called RECODE, for zero-shot VRD that addresses the limitations of traditional class-based prompts. It decomposes the visual features of a triplet into subject, object, and spatial features and generates detailed descriptions of visual cues for each relation category. The use of chain-of-thought prompting for generating reasonable weights is a unique and creative approach.\n\n2. The approach leverages large language models (LLMs), specifically GPT-3.5-turbo and CLIP, for the generation of descriptions and similarity calculations. The use of LLMs provides a strong foundation for generating informative and accurate descriptions of visual cues. The evaluation is conducted on four benchmark datasets, and the results demonstrate significant improvements over baseline methods.\n\n3. The paper provides clear descriptions and explanations of the proposed framework, including the visual feature decomposing, semantic feature decomposing, and relation classification steps. The process of generating descriptions of visual cues and weights using LLMs is well-described, and the chain-of-thought method is illustrated with examples. The evaluation metrics and experimental setup are clearly presented.\n\n4. The proposed approach addresses the challenge of zero-shot VRD by improving the discriminability of similar relation categories. By incorporating specific visual cues and generating descriptions, the approach enhances the performance of relation classification. The experimental results show significant improvements over baseline methods, demonstrating the effectiveness and interpretability of the proposed approach. The approach has the potential to advance the field of VRD and contribute to applications such as image understanding, scene understanding, and human-computer interaction.", "weaknesses": "While the experimental results show improvements over baseline methods, the paper lacks a thorough analysis of failure cases. Understanding when and why the proposed approach fails to accurately predict relations is crucial for identifying its limitations and potential areas of improvement. Analyzing failure cases and providing insights into the challenges faced by the model would strengthen the evaluation and guide future research directions.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We are willing to address all your questions.\n## Analysis of Failure Cases\nWe sincerely appreciate the reviewer's valuable feedback and the suggestion to include a thorough analysis of failure cases in our paper. We have conducted an in-depth examination of failure cases in our proposed method. As shown in Fig\\._R 2 (cf, **PDF file**), the descriptions of mismatch relations are consistent with the content of images (e.g., the armrests of the toilet), leading to mismatch yet *reasonable* predictions. We can observe certain scenarios where the relation between the subject and object could be interpreted in multiple ways, and both interpretations could be considered reasonable. For instance, in the case of the \"girl-toilet\", both \"using\" and \"sitting on\" are plausible given the context. However, the ground truth only contains one correct label. On the other hand, we also found instances where our method outperformed the ground truth annotations. For instance, in the case of \"man-snow\", our method accurately predicted the relation as \"lying on\" in Fig\\._R 2. This observation highlights the robustness of our approach.\n\n## Visual Cues Selection and Alternative\nIn our work, we focus on a **training-free** setting (cf, the **global response** for clarifying experimental settings), which directly involves inference for zero-shot VRD. Due to this difficult setting, there are two challenges: 1) we are unable to learn visual cues by designing training objectives. 2) we are unable to evaluate the quality of visual cues rigorously.\n\n- **Cues Selection.** To tackle these issues, we propose a specific approach for selecting visual cues, as outlined in Eq\\.(1). Specifically, for the subject, object, and spatial descriptions, we leverage the power of the GPT model to generate a random number of descriptions. Subsequently, we compute the similarity between these descriptions and their corresponding visual features, ultimately taking the mean of these similarity scores. Additionally, the weights assigned to the subject, object, and spatial location components are determined by the GPT model, which is detailed in Section 2.2.2.\n\n- **Alternative Visual Cues.** In our experiments, we have thoroughly considered the use of different prompts for generating visual cues, as explained in Section 2.2.1: (1) **Relation class description** (cf, Figure 4(a)) generates descriptions for each relation class directly. (2) **Relation component description** (cf, Figure 4(b)) generates descriptions for each component of the relation separately. (3) **Guided relation component description** (cf, Figure 4(c)) incorporates high-level object category to guide generation process. As shown in Table 1, our results demonstrate that the guided relation component description yields superior performance.\n\n## Performance Scale and Computational Limitations\nIn our proposed approach, the absence of a training phase makes the overall time consumption mainly driven by the inference process. As the number of related categories increases, the total visual descriptions over all relations also increases, the similarity computation between visual features and text features of these descriptions will contribute to the increase in time consumption (cf, Table\\_R 5). However, given all categories, the time consumption of similarity calculation is also unavoidable for other zero-shot work.\n\n## Evaluation Metrics\nThe evaluation metrics used in our work, such as Recall@K (R@K) and Mean Recall@K (mR@K) for SGG datasets (VG and GQA) [1] and mean Average Precision (mAP) for HOI datasets (HOI-DET and V-COCO) [2], are **widely adopted in the field of (zero-shot) visual relation detection**. However, we acknowledge that there may be certain limitations or biases associated with these metrics. Specifically, due to the influence of long-tail distribution in the datasets, the R@K may exhibit a bias towards the head predicate categories (categories with many samples) [1]. This bias could potentially affect the overall evaluation results and may not fully capture the model's performance on rare or less frequent relation categories. Thus, we also report m@R as a reference. Although these metrics are already widely used, designing good evaluation metrics for VRD itself is still an open problem.\n\n## Limitations\nWe have discussed the limitations in **Section F** in Supplementary Material.\n\n[1] Unbiased scene graph generation from biased training. In CVPR, 2020. \\\n[2] Exploring structure-aware transformer over interaction proposals for human-object interaction detection. In CVPR, 2022.\n"}, {"review_id": "OcIgUpdZs8", "reviewer": "Reviewer_q7Rt", "summary": "This paper proposed a novel method for zero-shot visual relation detection by leveraging LLM (e.g. GPT) and VLM (e.g. CLIP). Specifically, the proposed approach decomposes each predicate category into subject, object and spatial component and enrich each section with the help of LLMs, which can generate the description-based visual cues to help distinguish semantically similar concepts. Different visual cues are used to enhance discriminability from different perspectives, and the authors again use LLM to assign weights to different components for effective fusion. Extensive experiments on four different datasets are provided to demonstrate the effectiveness and interpretability.", "questions": "Overall I think the paper is solid just that the evaluation is insufficient, as it doesn't have any comparison with previous methods. I will consider raising my scores if the comparison is provided in a revised version (I only listed a few papers in the weaknesses sections and I'm pretty sure they are not the latest ones, please compare against the most recent one/SOTA methods as it is more meaningful),", "limitations": "This paper doesn't discuss the limitations of the proposed methods, for example one important underlying assumption is the reliability of the LLM (responsible for decomposition and weight estimation) and the quality of the VLM model. I don't see obvious potential negative societal impact with this paper.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed approach is theoretically sound and intuitive. Enriching the prompt from class-based to description-based can provide more information to enhance the relation sensitivity, and it also improves the explainability as the relation classification score can reveal the most important factors for the prediction.\n2. The decomposition of subject-object pair makes it much more efficient for processing visual signals as the previous O(N^2) patches now reduce to O(N). The spatial relationship also makes sense as an abstract from real objects to just the relations.\n3. The paper is well-written and easy to follow. The extensive experiments and ablation studies/visualization help a lot for understanding the model.", "weaknesses": "1. The most important issues with this paper is that the evaluation section does not have important baselines. Specifically, in Table 1 and Table 2 the authors only show the performance of the proposed model with simplified version (CLS and CLSDE), which more like an ablation study. Many previous work actually attempted similar tasks and have been experimenting on the same dataset, e.g. [1][2][3]. \n2. In Table 2 I guess the bolded numbers should be the highest (best)? For HICO-DET, CLS has the same performance on \"Rare\" category  with RECODE thus should be highlighted as well I think?\n3. A very very minor issue: the zero-shot chain-of-thought prompt used in most literatures are \"let's think step by step\" not \"let's think it step by step\". Formal usage should be \"think\" or \"think about it\" or \"think through it\", rather than \"think it\".\n\n[1] https://arxiv.org/abs/1804.10660\n[2] https://arxiv.org/abs/1707.09423v2\n[3] https://arxiv.org/pdf/2004.00436.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We are willing to address all your questions.\n## Comparison with More Baselines\n**Table_R2**: Comparison with SOTA VRD methods on the VG dataset. Note that none of these methods can be applied in the **training-free** zero-shot setting.\n\n| Model               | No Training | Unseen Relation | Training Data Source | zR@20 | zR@50 | zR@100 |\n|---------------------|-------------|-----------------|----------------------|-------|-------|--------|\n| Motifs [1]        | ❌          | ❌              | VG                   | 8.9   | 15.2  | 18.5   |\n| COACHER [2]       | ❌          | ❌              | VG & ConceptNet      | 28.2  | 34.1  | 37.2   |\n| DPL [3]           | ❌          | ❌              | VG                   | 6.0   | 7.7   | 9.3    |\n| CaCao [4]         | ❌          |    ✔️            | VG & CC3M & COCO     | 17.2  | 21.3  | 23.1   |\n| RECODE (ours)       |    ✔️   |      ✔️        |   --                  | 8.2   | 16.1  | 23.2   |\n\n- **Comparison with Training-based Methods.** Here we compared the proposed **training-free** RECODE framework with those well-designed training-based ones. Note that such comparisons are **unfair** as training-based frameworks can learn the underlying patterns and data distribution from the training set. For completeness, we still reported the results and investigate the performance gap between training-based frameworks and RECODE. Specifically, we compared the proposed RECODE with several relevant baselines, including triplet-level zero-shot VRD [1, 2], few-shot VRD [3], and category-level zero-shot VRD [4]. Since all of them cannot detect relations without training, we reported Zero-shot Recall@K (**zR@K**), which only calculates the Recall@K for those unseen **triplet** categories.\n\n  - Triplet-level zero-shot VRD methods:\n    - Motifs [1] is a traditional strong baseline without explicitly modeling the nature of zero-shot.\n    - COACHER [2] explicitly models the nature of zero-shot and takes the power of common sense from ConceptNet, resulting in better performance.\n  - Few-shot VRD methods:\n    - DPL [3] is a few-shot baseline, which mainly investigates making predictions with a few examples (here we evaluate 1-shot).\n  - Category-level zero-shot VRD methods:\n    - CaCao [4] also explicitly models the nature of zero-shot, and leverages language information from captions of CC3M and COCO for enhanced performance.\n\n  Surprisingly, even without training, RECODE still achieves competitive results, with zR@20, zR@50, and zR@100 of 8.2%, 16.1%, and 23.2%, respectively. This signifies its potential in handling unseen categories, due to the effective visual cues and inference mechanisms.\n\n- **Generalization on More Training-free Baselines.** Furthermore, we reported the results of our method applied in different SOTA visual-language models in Table_R 1, which also proves the effectiveness of RECODE.\n\n## Minor Error and Limitations\n\nWe appreciate your suggestions and will address the issues in the revised version. Besides, we have discussed the limitations in **Section F** in Supplementary Material.\n\n[1] Neural motifs: Scene graph parsing with global context. In CVPR, 2018. \\\n[2] Zero-shot scene graph relation prediction through commonsense knowledge integration. In ECML PKDD, 2021. \\\n[3] Decomposed prototype learning for few-shot scene graph generation.\tarXiv preprint arXiv:2303.10863, 2023. \\\n[4] Visually-prompted language model for fine-grained scene graph generation in an open world. In ICCV, 2023."}, {"review_id": "k19ky8oCux", "reviewer": "Reviewer_4NUf", "summary": "This paper aims to address the VRD problem using LLMs. The paper decomposes the visual features into human, object, and spatial features, and designs prompts to generate visual cues that describe each of these types of visual features. The relation classification is established by calculating the distance between visual and semantic features, and dynamic weights generated by LLMs are also integrated to enhance the training process.", "questions": "1. Could the authors provide additional examples of prompts and the corresponding visual cues generated by GPT that were utilized in the real experiments?\n2. Could the authors present more empirical evidence to further support the benefits of RECODE?", "limitations": "The authors have adequately discussed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. This paper focuses on a critical issue in visual relationship detection tasks, exploring the potential of leveraging LLMs to enhance visual relationship understanding.\n2. The paper introduces a novel and reasonable approach by decomposing each predicate category into human, object, and spatial descriptions.\n3. The authors thoroughly investigate various approaches to enhance the quality of prompts, encompassing both the generation of visual cues and the improvement of weights.", "weaknesses": "1. It appears that the visual cues employed in the main papers are presented as mere examples, leaving uncertainty regarding the specific visual cues utilized in the experiments. Furthermore, the visual cues depicted in Figures 3 and 4 exhibit notable differences, with Figure 4 generating more complex sentences. As a result, evaluating the quality of the visual cues based on the current evidence becomes challenging.\n2. The RECODE's performance gain on the HICO-DET and V-COCO datasets is marginal, and the authors did not provide error bars in their reports. Additionally, the ablation studies were solely conducted on the VG dataset, which could have substantial differences compared to the HICO-DET dataset. Consequently, it is difficult to be convinced that the RECODE is as effective as claimed by the authors.\n3. The main technical contribution of this paper lies in the development of specifically designed prompts. However, the improvements made to the prompts are relatively straightforward, and the utilization of CoT is a standard practice.\n4. There are many incurious statements/claims. For example, in line 47-55, it is unclear why a person has to stand while holding an object.; in lines 67-69, it is not clarified why the act of holding depends on spatial factors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We are willing to address all your questions.\n## Clarification of Visual Cues\n\n- **Examples in Figure 3 and Figure 4.**\nThe short prompts showcased in Figure 4 were primarily intended to demonstrate the enhanced accuracy achieved by the Guided Relation Component Description. In the real experiment, to standardize the output format, we carried out a **normalized visual cue description prompt** (cf, **Section B** in supplementary materials), as mentioned in footnote2 in Line 174. All of our experiments were conducted using these normalized prompts to maintain uniformity in our approach. We will revise this part to reduce misunderstanding.\n\n- **Extra visual cue examples.**\nWe would like to draw your attention to the fact that Figure 2, Figure 6, and Figure 7 already showcase a selection of generated visual cues derived from the normalized visual cue description prompt. Besides, Fig\\._R 2 (cf, **PDF file**) also shows extra visual cues generated by the normalized prompt.\n\n## Empirical Evidence\n\n- **Error Bars.**\nAs shown in Fig\\._R 3 (cf, **PDF file**), we plotted the error bars for three splits on the HICO-DET dataset. Across the three distinct splits - Full, Rare, and Non-Rare - our model's performance showcases impressive stability, i.e., 32.5% to 32.8%, 33.18% to 33.33%, and 32.2% to 32.55% for the Full, Rare, and Non-Rare split, respectively. These minuscule error bars reflect the high degree of consistency in our results, indicating that the measured values are reliable and repeatable.\n\n- **Ablation studies on HICO-DET and VCOCO datasets.** We have extended our ablation studies to encompass both the HICO-DET and VCOCO datasets. The results of these ablation studies are presented in Table_R 3, providing a more comprehensive evaluation of the effectiveness and generalizability of our proposed RECODE method.\n\n**Table_R 3**: Ablation studies on the HICO-DET and VCOCO datasets.\n| Cue | Spatial | Weight | HICO-DET (Full) | HICO-DET (Rare) | HICO-DET (Non-Rare) | VCOCO (Scenario 1) | VCOCO (Scenario 2) |\n|---|---|---|---|---|---|---|---|\n|   |   |   | 30.9 | 30.7 | 31.0 | 25.5 | 28.6 |\n|  ✔️ |   |   | 32.5 | 33.0 | 32.2 | 25.8 | 28.9 |\n|  ✔️ | ✔️ |   | 32.6 | 33.0 | 32.4 | 25.7 | 28.8 |\n| ✔️ |   |✔️ | 32.7 | 33.1 | 32.5 | 25.9 | 29.0 |\n| ✔️ | ✔️ | ✔️ | **32.7** | **33.2** | **32.5** | **26.0** | **29.0** |\n\n\n\n- **More comparisons with SOTA methods.** In Table_R 1 and Table_R 2, we also reported the results of combining different SOTA visual-language models and comparing with other SOTA SGG methods, which proves the effectiveness and universality of our method.\n\n## Contributions\nThank you for your thoughtful feedback on the main technical contribution of our paper. We appreciate the opportunity to clarify our focus and emphasize the key aspects of our work (cf, the **global response** for main contributions).\n\n- We point out that the challenges of relation sensitivity, spatial discriminability, and computational efficiency in zero-shot VRD. We achieve this by decomposing the task into three distinct components: subject, object, and spatial descriptions. Designing specific prompts is a feasible way to achieve this goal, which is not the central focus of our contribution.\n\n- We claim that the importance of each component description is different. Notably, we are the first work to utilize LLMs to effectively assign reasonable weights to these components. The CoT is employed solely to enhance the rationality of the weight generation process.\n\n## \"Incurious\" Statements\nRegarding the statement in Lines 47-55, we sincerely apologize for any confusion caused. Our intention was not to assert an absolute requirement for a person to stand while holding an object. Rather, we aimed to convey that in many observed scenarios, individuals are more commonly seen standing while holding objects, and we used the phrase \"**might be**\" in Lines 47-55 to indicate this likelihood. Similarly, we appreciate the need for clarity in the statement made in Lines 67-69. We would like to emphasize that spatial cues can play a crucial role in distinguishing certain relations, such as \"laying on\" and \"holding,\" where subject and object cues alone may not be sufficient for discrimination. We intend to revise this statement to better reflect our intent, underscoring the importance of spatial cues in the identification of specific relations.\n"}, {"review_id": "lkbXUDphSB", "reviewer": "Reviewer_EWRp", "summary": "In this paper, the authors developed a joint model of CLIP and LLM to solve the task of Visual relation detection. In this model, images are encoded into a triplet,~\\ie, object, subject and spatial branches. Then it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Experiments on four VRD benchmarks shows good results compared to the baseline models.", "questions": "Please refer to the weakness section.  \n\nBesides the technical novelty, this manuscript exist many unclear implementation problems and insufficient experimental results, especially a lack of discussion with other works using LLMs.\n\n I hope the authors could solve these concerns to make this paper more readable.", "limitations": "The authors clearly discussed the limitations of the proposed method. These discussions are fair.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ Interesting and good applications of LLMs including GPT-3.5 and large multi-modality pertaining models~\\eg, CLIP.\n+ Show higher performance than baseline methods.\n+ Good storytelling to readers understand the key idea.", "weaknesses": "### Technical Novelty and main ideas\n1. This paper pays its major attention to designing and using LLM~\\ie, GPT-3.5 to facilitate the deduction of multi-modal models. Many contributions lie in the design and feeding prompts into LLMs. This contribution seems insignificant to me and seems not generalizable for future LLM using different architectures with GPT-3.5. Besides, the authors also lack deep investigation into the improvements of the Chain of Thought (CoT).  Although the overall application and using LLMs seem interesting. but the solid contributions of this paper are not clear to me.\n\n### Presentation and motivation issues\nThe descriptions of CoT and experimental results are somehow unclear.  Besides, the novelty of using CoT in this task is not sufficient. Designing prompts \n\n2. The reviewers tested the same prompt using GPT-3.5, while in this case, with or without Chain of Thought (CoT) does not show significant differences. (with CoT: avg: 0.51, 0.3, 0.19 for s,o,p; w/o CoT: avg: 0.59, 0.33, 0.14 for s,o,p) The reviewer understands the results may not be stable but is still unclear about this case.\n\n3. Why did Figure 5(a) choose 0.4, 0.4, 0.2? The authors should explain how `obviously unreasonable` on lines 188 - 190 is `obviously unreasonable`.\n\n### Experimental issues\n\n4. For Figure 2 and Figure 7 in the supplementary material.\na) Where did the cues for the CLS baseline displayed by the sentence come from? Line 214 of the text states to use \"relational CLasS-based hints (e.g., ride)\", which are somehow inconsistent with the statements. Can the authors explain in more detail how the CLS baseline is tested? Where does the performance difference come from if two settings use the same prompts?\n5. Although the newly proposed setting, the authors do not discuss other similar works.  PEVL[1] and STIP[2]. Differences and relation discussion could help.\n\n[1] Yao, Y., Chen, Q., Zhang, A., Ji, W., Liu, Z., Chua, T. S., & Sun, M. (2022). PEVL: Position-enhanced pre-training and prompt tuning for vision-language models. _arXiv preprint arXiv:2205.11169_.\n\n[2] Zhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., & Chen, C. W. (2022). Exploring structure-aware transformer over interaction proposals for human-object interaction detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (pp. 19548-19557).\n\n6. The time efficiency of different compared methods in Tab.4 should also be provided.\n7. [Experimental Description] For the conjecture in line 247, does deduction give evidence for a confusion matrix? The reviewer did a cursory check of the dataset and there are a total of 117871 annotations, These provided actions seem to be rare.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed comments. We are willing to address all your questions.\n## Technical Novelty and Main Ideas\nWe appreciate your attention to prompt design for GPT-3.5. While the proposed prompts are tailored to GPT-3.5, the core idea of using compositional visual cues for VRD and utilizing prompts for weight allocation is a promising approach that can be extended to future LLMs with various architectures beyond GPT-3.5. Further elaboration on these contributions is available in the **global response**.\n## Presentation and Motivation Issues\nIn our study, we emphasize the varying importance of different components. Our main aim is to have LLMs assess the importance of subject, object, and spatial components. Reviewing examples from the reviewer, it's evident that the subject holds more importance than the object. To further validate this, we gathered statistics about the \"look at\" relation samples in VG dataset: 28 subject categories and 95 object categories. Such concentration on subjects implies their greater role, making the weights (0.4, 0.4, 0.2) for subject, object, and spatial components (equal weight for subject and object) appear less reasonable. We'll employ milder terms in place of \"obviously unreasonable''. \n\nAlthough the results of weight assignment without CoT may in some cases be similar to those with CoT, the motivation for integrating CoT is to enable LLMs to \"think before they act\". This requires LLMs to ask themselves: \"Why am I weighting this way?\", which in most cases leads to reasonable weights. As suggested in a concurrent work [1], GPT may be focusing on the last sentence \"The sum of weights must be 1.0!\", ignoring other important context. We also investigated the effect of introducing CoT, as shown in Table_R 4. The experimental results demonstrated that introducing CoT achieves consistent improvements, which solidly proves its effectiveness.\n\n**Table_R 4**: Comparison with or without CoT on the VG dataset.\n|CoT|R@20|R@50|R@100|mR@20|mR@50|mR@100|\n|-----|------|------|-------|-------|-------|--------|\n|❌|9.5|17.3|24.6|10.2|18.0|25.6|\n|✔️|10.6|18.3|25.0|10.7|18.7|27.8|\n## Experimental Issues\n- **Setting.** 1) *Cues for the CLS.* We'd like to clarify that the CLS baseline does not involve the utilization of visual cues (description-based prompts). 2) *CLS baseline.* In a manner similar to various previous object classification approaches, CLS employs relational-class-based prompts as discussed in Line 102. For instance, it adopts prompts like \"[REL-CLS]-ing/ed\", as illustrated by the example \"riding\" mentioned in Line 214. CLS obtains the classification score by calculating the similarity between the text feature of class-based prompt of each relation category and the visual feature. 3) *Difference.* In Figure 2 and Figure 7, the baseline \"CLIP\" (CLIP) relies solely on class-based prompts for generating predictions. While the method \"CLIP with Visual Cues\" (RECODE) takes advantage of both class-based prompts and description-based prompts.\n\n- **Discuss Similar Works.** Thank you for pointing out the relevance of discussing other similar works and providing references to PEVL, STIP, and other LLM work.\n  - *PEVL.* It focuses on proposing a new pre-training approach for vision-language models, positioning itself as a competitor to CLIP. On the other hand, our work is centered around training-free zero-shot settings, which require no training data and enable direct predictions without any fine-tuning.\n  - *STIP.* It emphasizes traditional fully-supervised methods for human-object interaction (HOI) detection, requiring training and evaluating on the same category set. In contrast, our approach focuses on training-free zero-shot settings, which do not rely on any training data.\n  - *Other LLM work.* The primary application of LLMs has predominantly focused on object classification. Directly extending these methods to visual relation detection (VRD) remains limited (cf, Line 301-302). Our work distinguishes itself as the pioneering effort to explore LLMs in the VRD domain, harnessing both LLMs and vision-language models (VLMs) to address VRD tasks efficiently, effectively, and with interpretability.\n\n  In the revised manuscript, we will include this comprehensive discussion.\n\n- **Time Efficiency.**\nWe investigated the time efficiency of each component in RECODE in Table_R 5. Specifically, we calculate the time required to infer each triplet and take the average. Regarding visual cues, their impact on latency remains marginal, amounting to a mere 14.5ms. In terms of spatial features, the computation of similarity demands 13ms. Notably, the spatial similarity in RECODE can be precomputed, given the spatial image within a finite set. Consequently, the calculation of spatial similarity doesn't lead to an increase in inference time, as it's retrieved from the precomputed list. It is pertinent to highlight that the weighting strategy inherent in RECODE does not encompass feature extraction, thereby resulting in a latency close to 0.\n\n**Table_R 5**: Analysis of key components on the VG dataset. Time (ms) represents the computation time of each triplet. The (•) represents the time when spatial component is retrieved offline.\n|Cue|Spatial|Weight|R@20|R@50|R@100|mR@20|mR@50|mR@100|Time(ms)|\n|---------------|---------|--------|------|------|-------|-------|-------|--------|-----------|\n||||7.2|10.9|13.2|9.4|14.0|17.6|46.7|\n|✔️|||7.4|12.3|16.6|9.0|14.0|19.5|61.2|\n|✔️|✔️||9.1|13.4|17.4|9.3|15.0|20.3|74.2(61.2)|\n|✔️||✔️|7.9|13.4|17.7|9.3|14.7|20.5|61.2|\n|✔️|✔️|✔️|**9.7**|**14.9**|**19.3**|**10.2**|**16.4**|**22.7**|74.2(61.2)|\n\n- **Confusion Matrix.**\nThe conjecture in Line 247 is verified in the confusion matrix. As shown in Fig\\._R4 (cf, **PDF file**), we found various similar appearances which may hinder the detection, e.g., \"eat at\" vs \"sit at\", \"ride\" vs \"straddle\", etc.\n\n[1] Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023."}]

Author rebuttal:
We appreciate the feedback from all reviewers. First of all, we would like to clarify and highlight our **different experimental settings** and **main contributions** over the existing work. Then, we will address all mentioned misunderstandings or questions from each reviewer individually.

## Different Experimental Settings

- **Existing Zero-Shot Settings.** For the Visual Relation Detection (VRD) task (or Scene Graph Generation, SGG), there are several different "zero-shot" evaluation settings. More specifically, let $\mathcal{O}^{train}$ ($\mathcal{O}^{test}$) and $\mathcal{R}^{train}$ ($\mathcal{R}^{test}$) be the set of object categories and relation categories during the training (test) stage, respectively. Meanwhile, we use $\mathcal{T}^{train}$ and $\mathcal{T}^{test}$ to denote relation triplet categories, which are the combinations of object and relation categories (e.g., "man-riding-bike") in the training and test set, respectively. Currently, all existing "zero-shot" VRD/SGG work can further be categorized into two types:

    - **Triplet-level Zero-shot VRD (with training)** [1]**.** In this setting, object and relation categories remain consistent across training and inference, i.e., $\mathcal{O}^{train} = \mathcal{O}^{test}$ and $\mathcal{R}^{train} = \mathcal{R}^{test}$, while certain triplet categories remain unseen by the model in the test set, i.e., $\mathcal{T}^{train} \ne \mathcal{T}^{test}$. Since the model has acquired knowledge about all objects and relations, the emphasis lies in evaluating its ability to generalize to these novel triplets within $\mathcal{T}^{test}$.
    - **Category-level Zero-shot VRD (with training)** [2, 3]**.** Here, both object and relation categories differ between training and inference phases, i.e., $\mathcal{O}^{train} \neq \mathcal{O}^{test}$ and $\mathcal{R}^{train} \neq \mathcal{R}^{test}$. In addition, the triplet categories are also different, i.e., $\mathcal{T}^{train} \neq \mathcal{T}^{test}$. To detect objects and relations from a much larger and potentially unlimited set of possible categories, the model should learn distinguishable and generalizable knowledge from the limited training set.

- **Our Training-free Zero-Shot Setting.** As above-mentioned, all existing zero-shot VRD/SGG work still *needs a training set for parameter learning*. In this work, we focus on a more challenging setting: **training-free zero-shot VRD, i.e., it solves VRD without any training stage**. Recently, the training-free paradigm has ushered in a new era to our community [4, 5, 6]. As for VRD, this new and challenging setting can notably reduce labor costs, particularly considering the complexities of manual labeling for relations [7]. On the other hand, it poses considerable challenges to perform such hard tasks without fine-tuning and labeled data.

The differences between our new "zero-shot" setting and existing work are also illustrated in Fig\._R 1 (cf, **PDF file**).

## Main Contributions

We understand the emphasis on the significance of our contributions might not have been sufficiently highlighted in the paper. We apologize for any confusion and would like to reiterate the main focus of our work:

- **Compositional Visual Cues for VRD**: Our primary contribution lies in the utilization of compositional visual cues to facilitate the challenging task of training-free zero-shot VRD. Instead of solely focusing on designing prompts to generate better descriptions or visual cues, we proposed the RECODE method, which leverages large language models (LLMs) to generate detailed and informative descriptions for different components of relation categories, such as subject, object, and spatial cues. These descriptions serve as description-based prompts that assist the vision-language pretrained models (e.g., CLIP) in distinguishing between similar relation categories and improving VRD performance.

- **Weight Assignment with LLMs and CoT**: In our work, we explored the importance of different components in relation categories, and recognized that their contributions are not equal. To address this, we introduced a novel approach using LLMs to assign reasonable weights for each component. The chain-of-thought (CoT) method was introduced as a guidance to generate rationales and weights, making the weight assignment more interpretable and reasonable. Note that CoT is not the main focus of our work; instead, it was employed as a tool to improve the quality and robustness of weight assignment.

We hope that these clarifications will better communicate the significance and novelty of our work. We thank the reviewers once again for their valuable feedback, which has helped us improve the quality and clarity of our paper.

[1] Zero-shot scene graph relation prediction through commonsense knowledge integration. In ECML PKDD, 2021.\
[2] Compositional prompt tuning with motion cues for open-vocabulary video relation detec- tion. In ICLR, 2023.\
[3] Visually-prompted language model for fine-grained scene graph generation in an open world. In ICCV, 2023.\
[4] Visual programming: Compositional visual reasoning without training. In CVPR, 2023.\
[5] Navgpt: Explicit reasoning in vision-and-language navigation with large language models. arXiv preprint arXiv:2305.16986, 2023. \
[6] Segment anything meets point tracking. arXiv preprint arXiv:2307.01197, 2023. \
[7] The devil is in the labels: Noisy label correction for robust scene graph generation. In CVPR, 2022.


Meta-review: This paper was reviewed by five experts in the field. The authors' rebuttal successfully resolved most of the concerns. Reviewers found the proposed method interesting and liked the presentation of the paper.

The AC agrees with the reviewers' assessments and believes the paper provides an interesting application of LLMs in-context learning to tackle a complex visual relationship detection (VRD) problem. Although the AC shares the major concern about weak baseline and limited improvements on existing VRD datasets HICO-DET and V-COCO, the overall novelty outweighs the concerns. The decision is to recommend the paper for acceptance. The reviewers did raise some valuable suggestions in the discussion that should be incorporated in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.

Why not spotlight / oral: 1) The proposed method is interesting but a straightforward application of LLMs to VRD; 2) the VRD performance is not strong enough on popular VRD datasets HICO-DET and V-COCO.
Decision: Accept (poster)