Title: Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Visual relation detection

Zero-short learning

Scene graph generation
Abstract: Pretrained vision-language models, such as CLIP, have demonstrated strong generalization capabilities, making them promising tools in the realm of zero-shot visual recognition. Visual relation detection (VRD) is a typical task that identifies relationship (or interaction) types between object pairs within an image. However, naively utilizing CLIP with prevalent class-based prompts for zero-shot VRD has several weaknesses, e.g., it struggles to distinguish between different fine-grained relation types and it neglects essential spatial information of two objects. To this end, we propose a novel method for zero-shot VRD: RECODE, which solves RElation detection via COmposite DEscription prompts. Specifically, RECODE first decomposes each predicate category into subject, object, and spatial components. Then, it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Different visual cues enhance the discriminability of similar relation categories from different perspectives, which significantly boosts performance in VRD. To dynamically fuse different cues, we further introduce a chain-of-thought method that prompts LLMs to generate reasonable weights for different visual cues. Extensive experiments on four VRD benchmarks have demonstrated the effectiveness and interpretability of RECODE.

Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models

Abstract

Pretrained vision-language models, such as CLIP, have demonstrated strong generalization capabilities, making them promising tools in the realm of zero-shot visual recognition. Visual relation detection (VRD) is a typical task that identifies relationship (or interaction) types between object pairs within an image. However, naively utilizing CLIP with prevalent class-based prompts for zero-shot VRD has several weaknesses, e . g ., it struggles to distinguish between different fine-grained relation types and it neglects essential spatial information of two objects. To this end, we propose a novel method for zero-shot VRD: RECODE , which solves RElation detection via COmposite DEscription prompts. Specifically, RECODE first decomposes each predicate category into subject, object, and spatial components. Then, it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Different visual cues enhance the discriminability of similar relation categories from different perspectives, which significantly boosts performance in VRD. To dynamically fuse different cues, we further introduce a chain-of-thought method that prompts LLMs to generate rea-

51.6%

vertical or horizontal subject above horizontal object

with a mid to large distance

CLIP

mounted on

riding

carrying

painted on

walking on

78.4%

19.8%

0.07%

0.05%

0.02%

CLIP with Visual Cues

riding

mounted on

watching

carrying

standing on

48.3%

0.1%

0.0%

0.0%

CLIP

holding

lying on

covered in

looking at

laying on

32.7%

25.4%

18.6%

16.4%

CLIP with Visual Cues

laying on

lying on

covering

looking at

sitting on

72.9%

27.0%

0.1%

0.0%

0.0%

riding

subject

object

spatial

with legs

with hands if holding reins or other equipment

seated on the back of the animal

with a body

with four legs

with a saddle or similar riding apparatus on its back

27.6

25.1

25.6

26.9

20.7

25.3

27.7

laying on

subject

object

spatial

with arms and legs stretched out

with limbs extended

with a flat surface

square or horizontal subject above horizontal

object with mid or large distance

with no movement

22.7

23.8

23.3

22.2

33.6

holding

subject

object

spatial

with hand

with arms

with a handle or a grip that is held by the human's hand

23.5

22.7

24.0

28.8

vertical human beside or under the square object with

a mid distance

man-riding-elephant

child-laying on-bed

𝜙

𝜙

𝜙

horizontal subject above horizontal

object with a mid distance

subject

object

spatial

with hands

with legs

with tool or equipment for mounting

with back or surface for mounting

with legs or hooves

20.7

23.8

26.7

25.5

21.8

25.0

mounted

on

𝜙

4.7%

Why

Why

Why

Why

sonable weights for different visual cues. Extensive experiments on four VRD benchmarks have demonstrated the effectiveness and interpretability of RECODE.

1 Introduction

Recent advances in pretrained vision-language models (VLMs) [1, 2, 3, 4] ( e . g ., CLIP [1]), have shown remarkable generalization ability and achieved impressive performance on zero-shot recognition tasks. Specifically, CLIP employs two encoders: an image encoder that converts images into visual features, and a text encoder that transforms sentences into semantic features. This design allows the encoders to map different modalities into a common semantic space. When the inputs to the text encoder are class-based prompts , such as 'A [CLASS]', 'A photo of [CLASS]', CLIP can compare the image and prompts in the shared semantic space, thereby enabling zero-shot recognition of novel categories [1]. Compared to object recognition, visual relation detection (VRD) is much more challenging, which needs to identify the relation types between object pairs within an image in the form of ⟨ subject , relation , object ⟩ [5, 6, 7, 8, 9]. It differs from object recognition in that it requires an understanding of how objects are related to each other. By crafting class-based prompts to describe these relation types, CLIP could potentially be extended to perform zero-shot VRD.

However, this straightforward baseline presents notable challenges. Imagine you are a child asked to distinguish relation categories 'holding' and 'carrying', both involving a person and an object. Based on the similar concepts of 'holding' ( i . e ., a person having an object in their hands) and 'carrying' ( i . e ., a person supporting an object in their hands), it would be difficult to determine the correct prediction ( cf. , Figure 1(a)). In other words, class-based prompts of 'holding' and 'carrying' might be projected to adjacent locations in semantic space by CLIP, leading to a relation sensitivity issue: CLIP struggles to differentiate between the subtle nuances of similar relations. Secondly, class-based prompts overlook the unique spatial cues inherent to each relation category, leading to a spatial discriminability issue. The 'holding' category generally suggests the object being at a certain height and orientation relative to the person, while 'carrying' implies a different spatial position, typically with the object located lower and possibly supported by the person's entire body. The neglect of spatial cues leads to inaccuracies in distinguishing between such spatial-aware relation categories. Moreover, applying CLIP in this manner brings about a computational efficiency issue. Using CLIP requires cropping each union region of a subject-object pair separately from the original image ( i . e ., N 2 crops for N proposals), leading to computational inefficiencies.

Nonetheless, we humans can distinguish relation categories from different visual cues. For example, from the subject's perspective, we could think that in the case of 'holding', a person might be standing while having an object, such as an umbrella, in their hand. Meanwhile, in the case of 'carrying', a person should be in a more engaged position, perhaps walking or running with both hands and arms supporting a heavy object, like a suitcase. In addition, spatial cues also play an important role in identifying these relation categories. For example, when a person is carrying an umbrella, the umbrella is usually positioned lower and closer to the person's body compared to when the person is holding an umbrella. Based on these visual cues, we can easily identify scenarios such as ' person-holding-umbrella ' and ' person-carrying-umbrella ' as in Figure 1(c).

Inspired by our humans' ability to extract and utilize different visual cues, we present a novel method for zero-shot VRD: RECODE , which classifies RElation via COmposite DEscriptions. It first uses large language models (LLMs) [10], to generate detailed and informative descriptions 2 for different components of relation categories, such as subject, object, and spatial. These descriptions are then used as description-based prompts for the CLIP model, enabling it to focus on specific visual features that help distinguish between similar relation categories and improve VRD performance. Specifically, for the subject and object components, these prompts include visual cues such as appearance ( e . g ., with leg), size ( e . g ., small), and posture ( e . g ., in a sitting posture). For the spatial component, these prompts include cues related to the spatial relationships between objects, such as relative position and distance. By incorporating different visual cues, RECODE enhances the discriminability of similar relation categories, such as 'riding' and 'mounted' based on the different postures of the subject, e . g ., 'seated on the back of animal' for the subject of 'riding'. Similarly, spatial visual cues can be used to differentiate between 'laying on' and 'holding' based on the relative position between the subject and object, such as 'subject above object' and 'subject under object' ( cf. , Figure 2).

In addition, we explore the limitations of several description generation prompts for visual cue, e . g ., relation class description prompt [11], and then design a guided relation component description prompt that utilizes the high-level object categories to generate more accurate visual cues for each relation category. For instance, if the high-level category of object is 'animal', the generated object descriptions for relation 'riding' are tailored to the 'animal' category, e . g ., 'with four legs', instead of the 'product', e . g ., 'with wheels'. Meanwhile, to better fuse the evidence from different visual cues, we further leverage LLMs to predict reasonable weights for different components. Particularly, we design a chain-of-thought (CoT) method [12] to break down this weight assignment problem into smaller, more manageable pieces, and prompt LLM to generate a series of rationales and weights.

To evaluate our RECODE, we conducted experiments on four benchmark datasets: Visual Genome (VG) [13] and GQA [14] datasets for scene graph generation (SGG), and HICO-DET [15] and V-COCO [16] datasets for human-object interaction (HOI) detection. Experimental results prove the generalization and interpretability of our method. In summary, we made three main contributions in this paper: 1) We analyze the weaknesses of the prevalent class-based prompt for zero-shot VRD in detail and propose a novel solution RECODE. RECODE leverages the power of LLMs to generate description-based prompts (visual cues) for each component of the relation class, enhancing the CLIP model's ability to distinguish between various relation categories. 2) We introduce a chain-of-thought method that breaks down the problem into smaller, more manageable pieces, allowing the LLM to generate a series of rationales for each cue, ultimately leading to reasonable weights for each component. 3) We conduct experiments on four benchmark datasets and demonstrate the effectiveness and interpretability of our method.

2 Approach

Typically, VRD is comprised of two sub-tasks: object detection and relation classification [5]. Since zero-shot object detection has been extensively studied [17, 1, 11], in this paper, we primarily focus on zero-shot relation classification. Specifically, given the bounding boxes (bboxes) { b i } and object categories { o i } of all objects, our target is to predict the visual relation (or predicate/interaction) categories { r ij } between pairwise objects. To facilitate presentation, we use s , o , and p to denote the subject, object, and their spatial position in a triplet respectively, and r to denote the relation category.

Class-based Prompt Baseline for Zero-Shot VRD. Following recent zero-shot object recognition methods, a straightforward solution for zero-shot VRD is the CLIP with class-based prompt. Specif-

riding

sitting on

looking at

laying on

flying in

…

woman-horse

Decomposing

subject

spatial

Image

Encoder

woman, with legs

woman, with hands

woman, seated on the back of

animal

d

:

d

𝒔𝟐

:

d

𝒔𝟑

:

horse, with a body

horse, with four legs

horse, with a saddle or similar

riding apparatus on its back

d

𝒐𝟏

:

d

𝒐𝟐

:

d

𝒐𝟑

:

vertical red box above

horizontal green box with a

mid distance

d

𝒑𝟏

:

Text

Encoder

Image

Encoder

Image

Encoder

Text

Encoder

v

𝑠

v

𝑜

v

𝑝

𝐭

𝑑𝑠2

Score

𝐭

𝑑𝑠1

𝐭

𝑑𝑠3

𝐭

𝑑𝑜2

𝐭

𝐭

𝑑𝑜3

𝐭

𝑑𝑜1

Text

Encoder

p

𝒄

:

riding

𝐭

𝑐

v

𝑠

v

𝑜

v

𝑝

𝐭

𝑑𝑠1

𝐭

𝑐

𝐭

𝑑𝑠2

𝐭

𝑑𝑠3

𝐭

𝑑𝑜1

𝐭

𝑑𝑜2

𝐭

𝑑𝑜2

𝐭

𝜙𝑑𝑠1

𝜙𝑐𝑠

𝜙𝑐𝑜

𝜙𝑑𝑠2

𝜙𝑑𝑠3

𝜙𝑑𝑜1

𝜙𝑑𝑜2

𝜙𝑑𝑜3

𝜙𝑑𝑝1

Visual Feature Decomposing

Semantic Feature Decomposing

Relation Classification

Text

Encoder

object

ically, a pretrained CLIP image encoder V ( · ) and a pretrained CLIP text encoder T ( · ) are used to classify pairwise objects with a set of relation classes. For each relation class, a natural language class-based prompt p c is generated, incorporating the relation information, e . g ., '[REL-CLS]-ing/ed' or 'a photo of [REL-CLS]'. Each prompt is then passed through T ( · ) to get semantic embedding t , while the union region of a subject-object pair is passed through V ( · ) to get visual embedding v . The cosine similarity between v and t of different relation categories is calculated and processed by a softmax function to obtain the probability distribution over all relation categories.

2.1 Zero-shot VRD with Composed Visual Cues

To overcome the limitations of class-based prompts, we propose a novel approach RECODE for zero-shot VRD. It consists of three parts: visual feature decomposing, semantic feature decomposing, and relation classification ( cf. , Figure 3). In the first two parts, we decompose the visual features of the triplet into subject, object, and spatial features, and then generate semantic features for each component. In the last part, we calculate the similarities between the decomposed visual features and a set of semantic features, and aggregate them to get the final predictions over all relations.

Visual Feature Decomposing. To enhance spatial discriminability and computational efficiency, we decompose the visual features of a triplet into subject, object, and spatial features. For subject and object features , we crop the regions of the subject and object from the original image using the given bboxes b s and b o , and encode them into visual embeddings v s and v o using the image encoder V ( · ) of CLIP. For spatial features , we aim to obtain the spatial relationship between the subject and object based on their bounding boxes. However, directly obtaining all spatial images based on the given bounding boxes is computationally expensive due to the diversity of spatial positions ( N 2 each image). To address this, we simulate the spatial relationship between the subject and object using a finite set of spatial images, represented by red and green bboxes respectively. We define four attributes (shape, size, relative position, and distance) based on bounding box properties. Each attribute is assigned a finite set of values to construct a finite set of simulated spatial images. For a given triplet, we match the calculated attribute values with the most similar simulated image 3 . The matched spatial image is then encoded into a visual embedding v p using V ( · ) of CLIP.

Semantic Feature Decomposing. To improve the CLIP model's ability to distinguish between different relation classes, we incorporate a set of description-based prompts D to augment the original class-based prompt for each relation category. For the subject and object components , we generate a set of description-based prompts D s and D o to provide additional visual cue information, the generation process is described in Sec. 2.2. These prompts contain object categories with specific visual cues that highlight the unique characteristics of the relation being performed, e . g ., 'women, with legs', which enhances the discriminability between similar relation categories. For the spatial component , it only contains a set of description-based prompts D p that include information about the relative position and distance between the subject and object in the image. By incorporating

What are useful features for

distinguishing a relation '

riding

'

in a photo?

What are the useful visual features

of

subject

and

object

that can

distinguish the relation '

riding

' in

a photo?

➢

The subject may be holding

onto reins or handles attached

to the object.

➢

The object may have a saddle

or other equipment specifically

designed for riding.

➢

……

(a) Relation Class Description

(b) Relation Component Description

What are the useful visual features of

subject and object that can distinguish

the relation '

riding

', when subject

belongs to '

human

' and object belongs

to '

animal

' ?

(c) Guided Relation Component Description

➢

The direction of movement

and speed of the person and

the mode of transportation.

➢

The surroundings, such as a

trail or arena, that suggest a

recreational or competitive

riding context.

➢

……

➢

The subject may be holding reins or

some sort of device used for

directing the animal's movement.

➢

The object may have a harness or

saddle on its body, which is often

used for riding or controlling the

animal.

➢

……

this additional information, we aim to distinguish between relations based on spatial location. After generating these sets of description-based prompts, we obtain semantic embeddings { t d s i } , { t d o i } , and { t d p i } using a text encoder T ( · ) , separately. These embeddings, along with the class-based prompt embedding t c , are used for relation classification.

Relation Classification. In this step, we compute the similarity score between the visual and semantic features to obtain the relation probability distribution. We first calculate the cosine similarity ϕ ( · , · ) between each visual embedding and semantic embedding for each relation category r . The final score incorporates both class-based and description-based prompts, and is calculated as follows:

where w k represents the importance of visual cues for each component k ∈ { s, o, p } , and | D k ( r ) | denotes the number of visual cues in D k ( r ) for relation category r . We compute the similarity of individual visual cues for each component and then obtain their average. The weights of different components are determined by a LLM, which will be discussed in Sec. 2.2. Finally, we apply a softmax operation to the scores to obtain the probability distribution over all relation categories.

2.2 Visual Cue Descriptions and Weights Generation

LLMs, such as GPT [10], have been shown to contain significant world knowledge. In this section, we present the process of generating descriptions of visual cue D s , D o , and D p , as well as the weights w s , w o , and w p for each component of each relation category using LLMs.

2.2.1 Visual Cue Descriptions

In this section, we explore methods for generating descriptions of visual cues for relation decomposition. Inspired by the work [11] of zero-shot image classification, we first propose relation class description prompt, which generates descriptions from the perspective of class-level ( cf. , Figure 4(a)). It has the advantage of producing descriptions that are easy to interpret and understand. However, it may result in overly diverse and information-rich descriptions that could hinder the extraction of meaningful visual cues, e . g ., 'speed of the person' in Figure 4(a).

To address this limitation, we then consider another relation component description prompt, which involves decomposing the relation into its subject and object components and generating descriptions of their visual features separately ( cf. , Figure 4(b)). While this type of prompt allows for more focused and specific descriptions of visual cues, it may not be effective in capturing the variations in visual features between different subject-object category pairs. For example, ' man-riding-horse ' and ' person-riding-bike ' typically have totally different visual features for the object. The visual cues 'reins' and 'saddle' of the object in Figure 4(b) are inappropriate for a 'bike'.

Suppose you are a relation classification model.

Given: subject belongs to 'human' and object belongs to

'product'.

The visual features of subject

: with eyes directed

towards the object, with head upright.

The visual features of object

: with visible features such

as front, display, or screen.

The visual features of position

: subject positioned either

above, below, left or right of the object at a mid distance.

Q: How do you weight these visual features to determine

the relation is 'looking at'?

The sum of weights must be

1.0

!

(a) Weight Prompt

➢

Visual features of subject: 0.4

➢

Visual features of object: 0.4

➢

Visual features of position: 0.2

· ·

· ·

· ·

The sum of weights must be 1.0

!

A

：

Let's think step by step

!

(CoT prompt)

(b) Weight Prompt with Chain-of-Thought

First, we need to consider the importance of the subject's

visual features. Since the direction of the eyes and head

position strongly indicate the focus of attention, we will give

them a weight of 0.6.

Next, we need to consider the importance of the object's

visual features. Since the visible features such as front,

display, or screen indicate that the object is something that

can be looked at, we will give them a weight of 0.3.

Finally,…, we will give them a weight of 0.1.

Therefore, we can weight these visual features as follows:

➢

Visual features of subject: 0.6

➢

Visual features of object: 0.3

➢

Visual features of position: 0.1

Therefore, we design guided relation component description prompt. It builds upon the second method by incorporating the high-level category information of the object into the generation process, leading to more accurate and informative descriptions of the visual features of both the subject and object components ( cf. , Figure 4(c)). To achieve this, we classify the object into high-level classes, such as 'human', 'animal', and 'product', to guide the description generation. For example, 'bike' is classified as 'product', and 'horse' is classified as 'animal'. This allows for the separate generation of visual feature descriptions for each high-level object class, e . g ., 'a harness or saddle on its body' for 'animal', resulting in more precise and relevant visual cues for each relation category 3 .

2.2.2 Visual Cue Weights

Intuitively, different combinations of visual cues may have varying degrees of importance in relation classification. For example, for relation 'looking at', the visual cue 'with visible features' of the object may not be as informative as the visual cue 'with eye' of the subject. To account for this, we leverage the impressive knowledge and reasoning abilities of LLMs to analyze the discriminative power of different visual cues and dynamically assign weights accordingly. Specifically, we provide each combination of visual cues as input to LLM and prompt it to determine the appropriate weight for each cue for distinguishing the given predicate. The prompts used for this purpose are in Figure 5.

Chain-of-Thought (CoT) Prompting. To ensure the generated weights are reasonable, we utilize a CoT method that has demonstrated remarkable reasoning abilities [12, 18]. Specifically, we prompt the LLM to generate rationales by using the stepwise reasoning prompt ' Let's think step by step! ' to break down the problem into smaller, more manageable pieces. Then LLM generates a series of rationales, and those that lead to the reasonable weights. For example in Figure 5, we demonstrate the importance of the CoT method in generating more accurate weights. Without the stepwise reasoning prompt, LLM generates the same weight for both the subject and object visual cues for 'looking at', which is clearly unreasonable. However, with the CoT prompt, LLM is able to analyze each cue step by step, leading to a more accurate assignment of weights, i . e ., the cues about the subject are relatively more important. In order to standardize the format of the strings generated by LLMs for extracting different components of visual cues and weights, we make certain modifications to the prompts for descriptions and weights 3 .

3 Experiment

3.1 Experiment setup

Datasets. We evaluated our method on four zero-shot VRD benchmarks: 1) VG [13] contains 26,443 images for testing, each annotated with object and predicate labels to form a scene graph. Following previous works [5], we used the pre-processed VG with 150 object classes. We adopted the 24 semantic predicate classes proposed in [19, 20], as they are more informative and challenging for classifying. 2) GQA [14] is a large-scale SGG dataset. We used the same split provided by [21], which contains 8,208 images for testing with 200 object classes. As for predicate classes, we selected

26 semantic predicate classes by referring to VG. 3) HICO-DET [15] contains 9,658 testing images annotated with 600 HOI triplets derived from combinations of 117 verb classes and 80 object classes. 4) V-COCO [16] comprises 4,946 testing images annotated with 29 action categories.

Evaluation Metrics. For SGG datasets ( i . e ., VG and GQA), we reported Recall@K (R@K) which indicates the proportion of ground-truths that appear among the top-K confident predictions, and mean Recall@K (mR@K) which averages R@K scores calculated for each category separately [22]. For HOI datasets ( i . e ., HOI-DET and V-COCO), we reported mean Average Precision (mAP) [23].

Implementation Details. For the LLM, we employed the GPT-3.5-turbo, a highly performant variant of the GPT model. As for CLIP, we leveraged the OpenAI's publicly accessible resources, specifically opting for the Vision Transformer with a base configuration (ViT-B/32) as default backbone 3 .

Settings. The bounding box and category of objects were given in all experiments. We compared our RECODE with two baselines: 1) CLS , which uses relation-CLasS-based prompts ( e . g ., 'riding') to compute the similarity between the image and text. 2) CLSDE , which uses prompts of relation CLasS DEscription as shown in Figure 4(a). Each component of the proposed framework can serve as a plug-and-play module for zero-shot VRD. Specifically, 1) Filter , which denotes filtering those unreasonable predictions ( e . g ., kid-eating-house ) with the rules generated by GPT 3 . 2) Cue , which denotes using description-based prompts (Sec. 2.1). 3) Spatial , which denotes using spacial images as additional features. 4) Weight , which denotes using dynamic weights generated by GPT to determine the importance of each feature, i . e ., visual cue weights.

3.2 Results and Analysis

In this work, we evaluated the prediction performance of the proposed framework on two related tasks, i . e ., SGG and HOI. The former outputs a list of relation triplet ⟨ sub,pred,obj ⟩ , while the latter just fix the category of sub to human. Overall, our method achieved significant improvement on the two tasks compared to the CLS baseline, which shows the superiority of our method.

Evaluation on SGG. From the results in Table 1, we have the following observations: 1) CLSDE showed worse performance than the trivial CLS baseline. This is because the considerable noise in CLSDE which may hinder the model to attend the most distinguishable parts. 2) With the proper guidance, RECODE achieved considerable improvements compared to the baselines, e . g ., 0.8% to 6.1% gains on VG and 0.7% to 2.9% gains on GQA. The

performance drops of RECODE † also demonstrated the importance of guidance from high-level object categories during the generation process. 3) Integrated with the filtering strategy, i . e ., RECODE ⋆ , achieved the best performance over all metrics, which suggests that commonsense knowledge is complementary and effective for zero-shot VRD. It also demonstrated that CLIP struggles to distinguish abstract concepts, i . e ., relation sensitivity as mentioned in Sec. 1.

Evaluation on HOI. Since standard evaluation procedure of HOI had filtered out those unreasonable predictions, RECODE ⋆ was not evaluated here. From the results in Table 2, we can observe that the performance gains were lower than those on SGG, e . g ., 0.0% to 0.7% gains on HICO-DET and 0.4% to 0.5% gains on V-COCO. The reasons are two-fold. On the one hand, since the category of subject is always a human, its features are too similar to be distinguished by CLIP. On the other hand, some of the actions are very similar in appearance. For example, distinguishing between actions like ' person-throw-sports ball ' and ' person-catch-sports ball ' is challenging due to their visual similarity.

3.3 Diagnostic Experiment

Architectures. We investigated the impact of changing the architectures of CLIP, as shown in Table 3. From the results, we can observe consistent improvements regardless of the architecture used.

Key Component Analysis. The results are summarized in Table 4. The first row refers to the CLS baseline. Four crucial conclusions can be drawn. First , with the guidance of Cue, consistent improvements can be observed, e . g ., 0.2% to 3.4% gains on R@K w/o Filter and 1.3% to 4.1% gains on R@K with Filter. Second , by introducing the spatial feature, the relative position of subject and object is considered, resulting in notable perfor-

mance gains on R@K (0.8% to 1.7%) and mR@K (0.3% to 1.0%) w/o Filter compared to just using Cue. This is because the spatial feature is of importance for relation detection [6]. Third , benefiting from the impressive reasoning ability of LLMs, the proposed weighting strategy can determine the importance of different cues, thus achieving further improvements, e . g ., 0.5% to 1.1% gains on R@K compared to average aggregation. Fourth , by filtering those unreasonable predictions, consistent improvements can be observed. The reason may be that the performance of relation detection of CLIP is not accurate enough. Empirically, commonsense knowledge is a feasible way to filter those noise. Combining all components allows for the best overall performance on all evaluation metrics.

Case study. To investigate the most important regions for distinguishing relations, we visualized the attention map given different images and prompts ( cf. , Figure 6). From the visualization of class-based prompts, we can observe that CLIP may attend those regions unrelated to the query prompts, e . g ., focusing on the body of a person given relation 'growing on'. We attribute this phenomenon to the insufficient information within given prompts, which is also our motivation to introduce visual cue descriptions. As for description-based prompts, CLIP can attend to right regions with the guidance of descriptions, e . g ., focusing on colorful patterns on the product given relation 'painted on'.

eating:

mouth toward the food

hanging from

: giraffe, with

limbs

hanging from

𝜙: 27.2

hanging from

𝜙: 30.8

growing on

: tree, a larger,

stationary tree with a sturdy

structure that can accommodate

the growing product

hanging from

: tree, with

a hook, loop or handle

growing on

𝜙: 23.2

growing on

𝜙: 27.3

hanging from

𝜙: 23.4

hanging from

𝜙: 24.5

using

𝜙: 23.4

looking at

𝜙: 25.2

using

𝜙: 25.2

looking at

𝜙: 23.3

using

: person, possibly with

fingers actively interacting with

something

looking at

: person, with

eyes directed towards

something

painted on

: vase, the painted

design or image may cover all

or part of the its body

mounted on

: vase, with

attachment mechanism

painted on

𝜙: 24.4

mounted on

𝜙: 26.4

painted on

𝜙: 32.3

mounted on

𝜙: 29.5

giraffe-eating

-leaf

person-using

-phone

leaf-

growing on-tree

flower-

painted on-vase

Description-based Prompts

Class-based Prompts

eating

𝜙: 26.1

eating

𝜙: 33.1

Subject Description:

animal-product

Subject Description:

human-product

Object Description:

product-product

Object Description:

product-product

4 Related Work

Visual Relation Detection (VRD) aims to predict the relationships of given subject-object pairs, which can be viewed as a pair-wise classification task and have been widely studied in the image domain, e . g ., scene graph generation (SGG) [5, 6, 22, 24] and human-object interaction (HOI) detection [25, 26, 27]. Previous solutions mainly focus on learning representations from the training samples on pre-defined categories, which may suffer from noisy annotations [22] or long-tailed predicate distribution [6, 28] and are far from the needs of the real-world scenarios. Recently, some attempts [8, 29] adopted prompt-tuning [30] to predict unseen categories during inference. However, since the learnable prompts may be overfitting when trained on seen categories, their performance is sensitive to the split of seen/unseen categories [31]. In contrast, our method can predict the relationships directly without any training samples, and has better interpretability and generalization ability, especially in rare informative relation categories.

Zero-shot Visual Recognition enables the model to recognize new categories that it has never seen during training, which is one of the research hotspots in the vision community. Aligning visual representations to pre-trained word embeddings ( e . g ., Word2Vec [32] and GloVe [33]) is an intuitive and feasible way to achieve this goal [34]. More recently, VLMs, which use contrastive learning [35] to learn a joint space for vision and language, have demonstrated their impressive zero-shot ability [1]. Therefore, many zero-shot works [36, 37, 38] adopted such VLMs as their basic component to use the knowledge of the learned joint space. However, most of them only utilized the class name of unseen categories during inference, which makes an over-strong assumption that the text encoder project proper embeddings with only category names [11]. Then, Menon and Vondrick [11] proposed to query LLMs for the rich context of additional information. Nonetheless, it is non-trivial to apply such paradigms to VRD as discussed in Sec. 1. To the best of our knowledge, we are the first to leverage both LLMs and VLMs for VRD in an efficient, effective, and explainable way.

5 Conclusion

In this paper, we proposed a novel approach for zero-shot Visual Relationship Detection (VRD) that leverages large language models (LLMs) to generate detailed and informative descriptions of visual cues for each relation category. The proposed method addresses the limitations of traditional class-based prompts and enhances the discriminability of similar relation categories by incorporating specific visual cues. Moreover, we introduced a chain-of-thought method that breaks down the problem into smaller, more manageable pieces, allowing the LLM to generate a series of rationales for each visual cue and ultimately leading to reasonable weights. Our experiments on four benchmark datasets demonstrated the effectiveness and interpretability of our method.

Acknowledgement. This work was supported by the National Key Research & Development Project of China (2021ZD0110700), the National Natural Science Foundation of China (U19B2043, 61976185), and the Fundamental Research Funds for the Central Universities (226-2023-00048). Long Chen is supported by HKUST Special Support for Young Faculty (F0927), and HKUST Sports Science and Technology Research Grant (SSTRG24EG04).

References

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , pages 8748-8763, 2021.

Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR , pages 7959-7971, 2022.

Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. Fine-grained image captioning with clip reward. In Findings of NAACL , 2022.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , pages 4904-4916, 2021.

Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR , pages 5410-5419, 2017.

Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In CVPR , pages 3716-3725, 2020.

Xingchen Li, Long Chen, Jian Shao, Shaoning Xiao, Songyang Zhang, and Jun Xiao. Rethinking the evaluation of unbiased scene graph generation. In BMVC , 2022.

Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Towards open-vocabulary scene graph generation with prompt-based finetuning. In ECCV , 2022.

Lin Li, Guikun Chen, Jun Xiao, Yi Yang, Chunping Wang, and Long Chen. Compositional feature augmentation for unbiased scene graph generation. In ICCV , pages 21685-21695, 2023.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NIPS , 2020.

Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR , 2022.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NIPS , 2022.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV , 2017.

Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR , pages 6700-6709, 2019.

Yu-Wei Chao, Zhan Wang, Yugeng He, Jiaxuan Wang, and Jia Deng. Hico: A benchmark for recognizing human-object interactions in images. In ICCV , pages 1017-1025, 2015.

Saurabh Gupta and Jitendra Malik. Visual semantic role labeling. arXiv preprint arXiv:1505.04474 , 2015.

Caixia Yan, Xiaojun Chang, Minnan Luo, Huan Liu, Xiaoqin Zhang, and Qinghua Zheng. Semantics-guided contrastive network for zero-shot object detection. TPAMI , 2022.

Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493 , 2022.

Neural motifs: Scene graph parsing with global context. In CVPR , 2018.

Anh Duc Bui, Soyeon Caren Han, and Josiah Poon. Sg-shuffle: Multi-aspect shuffle transformer for scene graph generation. In Australasian Joint Conference on Artificial Intelligence , pages 87-101. Springer, 2022.

Xingning Dong, Tian Gan, Xuemeng Song, Jianlong Wu, Yuan Cheng, and Liqiang Nie. Stacked hybrid-attention and group collaborative learning for unbiased scene graph generation. In CVPR , pages 19427-19436, 2022.

Lin Li, Long Chen, Yifeng Huang, Zhimeng Zhang, Songyang Zhang, and Jun Xiao. The devil is in the labels: Noisy label correction for robust scene graph generation. In CVPR , pages 18869-18878, 2022.

Yu-Wei Chao, Yunfan Liu, Xieyang Liu, Huayi Zeng, and Jia Deng. Learning to detect human-object interactions. In WACV , pages 381-389, 2018.

Yao Teng and Limin Wang. Structured sparse r-cnn for direct scene graph generation. In CVPR , pages 19437-19446, 2022.

Keizo Kato, Yin Li, and Abhinav Gupta. Compositional learning for human object interaction. In ECCV , pages 234-251, 2018.

Dong-Jin Kim, Xiao Sun, Jinsoo Choi, Stephen Lin, and In So Kweon. Detecting human-object interactions with action co-occurrence priors. In ECCV , pages 718-736, 2020.

Yue Liao, Aixi Zhang, Miao Lu, Yongliang Wang, Xiaobo Li, and Si Liu. Gen-vlkt: Simplify association and enhance interaction understanding for hoi detection. In CVPR , pages 2012320132, 2022.

Guikun Chen, Lin Li, Yawei Luo, and Jun Xiao. Addressing predicate overlap in scene graph generation with semantic granularity controller. In ICME , pages 78-83. IEEE, 2023.

Kaifeng Gao, Long Chen, Hanwang Zhang, Jun Xiao, and Qianru Sun. Compositional prompt tuning with motion cues for open-vocabulary video relation detection. In ICLR , 2023.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Mingfei Gao, Chen Xing, Juan Carlos Niebles, Junnan Li, Ran Xu, Wenhao Liu, and Caiming Xiong. Open vocabulary object detection with pseudo bounding-box labels. In ECCV , pages 266-282, 2022.

Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013.

Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP , pages 1532-1543, 2014.

Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR , pages 6857-6866, 2018.

Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In NIPS , 2020.

Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR , pages 14393-14402, 2021.

Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR , 2022.

Tao Wang and Nan Li. Learning to detect and segment for open vocabulary object detection. In CVPR , 2023.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors developed a joint model of CLIP and LLM to solve the task of Visual relation detection. In this model, images are encoded into a triplet,~\\ie, object, subject and spatial branches. Then it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Experiments on four VRD benchmarks shows good results compared to the baseline models.\n\nSTRENGTHS:\n+ Interesting and good applications of LLMs including GPT-3.5 and large multi-modality pertaining models~\\eg, CLIP.\n+ Show higher performance than baseline methods.\n+ Good storytelling to readers understand the key idea.\n\nWEAKNESSES:\n### Technical Novelty and main ideas\n1. This paper pays its major attention to designing and using LLM~\\ie, GPT-3.5 to facilitate the deduction of multi-modal models. Many contributions lie in the design and feeding prompts into LLMs. This contribution seems insignificant to me and seems not generalizable for future LLM using different architectures with GPT-3.5. Besides, the authors also lack deep investigation into the improvements of the Chain of Thought (CoT).  Although the overall application and using LLMs seem interesting. but the solid contributions of this paper are not clear to me.\n\n### Presentation and motivation issues\nThe descriptions of CoT and experimental results are somehow unclear.  Besides, the novelty of using CoT in this task is not sufficient. Designing prompts \n\n2. The reviewers tested the same prompt using GPT-3.5, while in this case, with or without Chain of Thought (CoT) does not show significant differences. (with CoT: avg: 0.51, 0.3, 0.19 for s,o,p; w/o CoT: avg: 0.59, 0.33, 0.14 for s,o,p) The reviewer understands the results may not be stable but is still unclear about this case.\n\n3. Why did Figure 5(a) choose 0.4, 0.4, 0.2? The authors should explain how `obviously unreasonable` on lines 188 - 190 is `obviously unreasonable`.\n\n### Experimental issues\n\n4. For Figure 2 and Figure 7 in the supplementary material.\na) Where did the cues for the CLS baseline displayed by the sentence come from? Line 214 of the text states to use \"relational CLasS-based hints (e.g., ride)\", which are somehow inconsistent with the statements. Can the authors explain in more detail how the CLS baseline is tested? Where does the performance difference come from if two settings use the same prompts?\n5. Although the newly proposed setting, the authors do not discuss other similar works.  PEVL[1] and STIP[2]. Differences and relation discussion could help.\n\n[1] Yao, Y., Chen, Q., Zhang, A., Ji, W., Liu, Z., Chua, T. S., & Sun, M. (2022). PEVL: Position-enhanced pre-training and prompt tuning for vision-language models. _arXiv preprint arXiv:2205.11169_.\n\n[2] Zhang, Y., Pan, Y., Yao, T., Huang, R., Mei, T., & Chen, C. W. (2022). Exploring structure-aware transformer over interaction proposals for human-object interaction detection. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (pp. 19548-19557).\n\n6. The time efficiency of different compared methods in Tab.4 should also be provided.\n7. [Experimental Description] For the conjecture in line 247, does deduction give evidence for a confusion matrix? The reviewer did a cursory check of the dataset and there are a total of 117871 annotations, These provided actions seem to be rare.\n\nQUESTIONS:\nPlease refer to the weakness section.  \n\nBesides the technical novelty, this manuscript exist many unclear implementation problems and insufficient experimental results, especially a lack of discussion with other works using LLMs.\n\n I hope the authors could solve these concerns to make this paper more readable.\n\nLIMITATIONS:\nThe authors clearly discussed the limitations of the proposed method. These discussions are fair.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to address the VRD problem using LLMs. The paper decomposes the visual features into human, object, and spatial features, and designs prompts to generate visual cues that describe each of these types of visual features. The relation classification is established by calculating the distance between visual and semantic features, and dynamic weights generated by LLMs are also integrated to enhance the training process.\n\nSTRENGTHS:\n1. This paper focuses on a critical issue in visual relationship detection tasks, exploring the potential of leveraging LLMs to enhance visual relationship understanding.\n2. The paper introduces a novel and reasonable approach by decomposing each predicate category into human, object, and spatial descriptions.\n3. The authors thoroughly investigate various approaches to enhance the quality of prompts, encompassing both the generation of visual cues and the improvement of weights.\n\nWEAKNESSES:\n1. It appears that the visual cues employed in the main papers are presented as mere examples, leaving uncertainty regarding the specific visual cues utilized in the experiments. Furthermore, the visual cues depicted in Figures 3 and 4 exhibit notable differences, with Figure 4 generating more complex sentences. As a result, evaluating the quality of the visual cues based on the current evidence becomes challenging.\n2. The RECODE's performance gain on the HICO-DET and V-COCO datasets is marginal, and the authors did not provide error bars in their reports. Additionally, the ablation studies were solely conducted on the VG dataset, which could have substantial differences compared to the HICO-DET dataset. Consequently, it is difficult to be convinced that the RECODE is as effective as claimed by the authors.\n3. The main technical contribution of this paper lies in the development of specifically designed prompts. However, the improvements made to the prompts are relatively straightforward, and the utilization of CoT is a standard practice.\n4. There are many incurious statements/claims. For example, in line 47-55, it is unclear why a person has to stand while holding an object.; in lines 67-69, it is not clarified why the act of holding depends on spatial factors.\n\nQUESTIONS:\n1. Could the authors provide additional examples of prompts and the corresponding visual cues generated by GPT that were utilized in the real experiments?\n2. Could the authors present more empirical evidence to further support the benefits of RECODE?\n\nLIMITATIONS:\nThe authors have adequately discussed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a novel method for zero-shot visual relation detection by leveraging LLM (e.g. GPT) and VLM (e.g. CLIP). Specifically, the proposed approach decomposes each predicate category into subject, object and spatial component and enrich each section with the help of LLMs, which can generate the description-based visual cues to help distinguish semantically similar concepts. Different visual cues are used to enhance discriminability from different perspectives, and the authors again use LLM to assign weights to different components for effective fusion. Extensive experiments on four different datasets are provided to demonstrate the effectiveness and interpretability.\n\nSTRENGTHS:\n1. The proposed approach is theoretically sound and intuitive. Enriching the prompt from class-based to description-based can provide more information to enhance the relation sensitivity, and it also improves the explainability as the relation classification score can reveal the most important factors for the prediction.\n2. The decomposition of subject-object pair makes it much more efficient for processing visual signals as the previous O(N^2) patches now reduce to O(N). The spatial relationship also makes sense as an abstract from real objects to just the relations.\n3. The paper is well-written and easy to follow. The extensive experiments and ablation studies/visualization help a lot for understanding the model.\n\nWEAKNESSES:\n1. The most important issues with this paper is that the evaluation section does not have important baselines. Specifically, in Table 1 and Table 2 the authors only show the performance of the proposed model with simplified version (CLS and CLSDE), which more like an ablation study. Many previous work actually attempted similar tasks and have been experimenting on the same dataset, e.g. [1][2][3]. \n2. In Table 2 I guess the bolded numbers should be the highest (best)? For HICO-DET, CLS has the same performance on \"Rare\" category  with RECODE thus should be highlighted as well I think?\n3. A very very minor issue: the zero-shot chain-of-thought prompt used in most literatures are \"let's think step by step\" not \"let's think it step by step\". Formal usage should be \"think\" or \"think about it\" or \"think through it\", rather than \"think it\".\n\n[1] https://arxiv.org/abs/1804.10660\n[2] https://arxiv.org/abs/1707.09423v2\n[3] https://arxiv.org/pdf/2004.00436.pdf\n\nQUESTIONS:\nOverall I think the paper is solid just that the evaluation is insufficient, as it doesn't have any comparison with previous methods. I will consider raising my scores if the comparison is provided in a revised version (I only listed a few papers in the weaknesses sections and I'm pretty sure they are not the latest ones, please compare against the most recent one/SOTA methods as it is more meaningful),\n\nLIMITATIONS:\nThis paper doesn't discuss the limitations of the proposed methods, for example one important underlying assumption is the reliability of the LLM (responsible for decomposition and weight estimation) and the quality of the VLM model. I don't see obvious potential negative societal impact with this paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents RECODE, a novel method for zero-shot visual relation detection (VRD), designed to address the shortcomings of models like CLIP in distinguishing subtle relation categories and spatial discriminability. RECODE leverages large language models (LLMs) to generate detailed description-based prompts for each relation class component, thereby enhancing VRD performance. The authors also introduce a chain-of-thought method that breaks down the problem into smaller parts for LLMs, thereby assigning reasonable weights for each component. The effectiveness and interpretability of the method are demonstrated through experiments on four benchmark datasets.\n\nSTRENGTHS:\n1. The approach introduces a novel framework, called RECODE, for zero-shot VRD that addresses the limitations of traditional class-based prompts. It decomposes the visual features of a triplet into subject, object, and spatial features and generates detailed descriptions of visual cues for each relation category. The use of chain-of-thought prompting for generating reasonable weights is a unique and creative approach.\n\n2. The approach leverages large language models (LLMs), specifically GPT-3.5-turbo and CLIP, for the generation of descriptions and similarity calculations. The use of LLMs provides a strong foundation for generating informative and accurate descriptions of visual cues. The evaluation is conducted on four benchmark datasets, and the results demonstrate significant improvements over baseline methods.\n\n3. The paper provides clear descriptions and explanations of the proposed framework, including the visual feature decomposing, semantic feature decomposing, and relation classification steps. The process of generating descriptions of visual cues and weights using LLMs is well-described, and the chain-of-thought method is illustrated with examples. The evaluation metrics and experimental setup are clearly presented.\n\n4. The proposed approach addresses the challenge of zero-shot VRD by improving the discriminability of similar relation categories. By incorporating specific visual cues and generating descriptions, the approach enhances the performance of relation classification. The experimental results show significant improvements over baseline methods, demonstrating the effectiveness and interpretability of the proposed approach. The approach has the potential to advance the field of VRD and contribute to applications such as image understanding, scene understanding, and human-computer interaction.\n\nWEAKNESSES:\nWhile the experimental results show improvements over baseline methods, the paper lacks a thorough analysis of failure cases. Understanding when and why the proposed approach fails to accurately predict relations is crucial for identifying its limitations and potential areas of improvement. Analyzing failure cases and providing insights into the challenges faced by the model would strengthen the evaluation and guide future research directions.\n\nQUESTIONS:\n1. The paper mentions the incorporation of specific visual cues to improve the discriminability of related categories. Could you provide more details on how these cues were selected? What criteria were used to determine their relevance and effectiveness? Additionally, did you consider any alternative visual cues during the experimentation process? Exploring different visual cues and discussing their impact could provide further insights into the effectiveness of the proposed approach.\n\n2. The paper demonstrates improvements in relation classification for zero-shot VRD, but it would be valuable to discuss the scalability of the proposed approach. How does the performance scale with an increasing number of related categories and visual concepts? Are there any computational or efficiency limitations that arise when dealing with larger datasets or more complex scenes? \n\n3. The paper should provide a clear justification for the choice of evaluation metrics used to assess the performance of the proposed approach. Are there any limitations or biases associated with the selected metrics? Additionally, it would be helpful to include a discussion on the limitations of these metrics in capturing the true performance of zero-shot VRD models.\n\n4. While the paper discusses the experimental results and improvements over baseline methods, it would be valuable to have a section dedicated to the limitations of the proposed approach. Identifying and addressing these limitations can help guide future research directions. Additionally, it would be beneficial to have a discussion on potential extensions or improvements to the proposed approach that could further enhance its performance or broaden its applicability.\n\nLIMITATIONS:\n1. Auhors has not included the section or discussion on limitations and, if applicable, the potential negative social impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
Naively utilizing CLIP with prevalent class-based prompts for zero-shot VRD has several weaknesses, e.g., it struggles to distinguish between fine-grained relation types and neglects essential spatial information of two objects. To this end, the authors propose a novel method for zero-shot VRD: RECODE, which solves RElation detection via COmposite DEscription prompts. Specifically, RECODE first decomposes each predicate category into subject, object, and spatial components. Then, it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Different visual cues enhance the discriminability of similar relation categories from different perspectives, boosting performance in VRD. To dynamically fuse different cues, they introduce a chain-of-thought method that prompts LLMs to generate reasonable weights for different visual cues.

STRENGTHS:
- The framework for decomposing visual cues and using LLM to separately generate prompts for subject, object, and spatial features seems novel.
- The proposed method shows noticeable performance improvements, and the authors provided an ablation study to solidly analyze the design choices of the proposed method.

WEAKNESSES:
- The baselines in the experiments seem weak. Are the baseline methods recent enough models? To verify the effectiveness of the proposed method, the RECODE should be attached to the recent state-of-the-art model and show consistent performance improvement.

QUESTIONS:
Please refer to the questions in the weakness.

LIMITATIONS:
I cannot find a potential negative societal impact in this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-07