SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Lin Li $^{1,2}$ , Jun Xiao $^{1}$ , Guikun Chen $^{1}$ , Jian Shao $^{1}$ , Yueting Zhuang $^{1}$ , Long Chen $^{2*}$ $^{1}$ Zhejiang University  $^{2}$ The Hong Kong University of Science and Technology  
{mukti, junx, guikun.chen, jshao, yzhuang}@zju.edu.cn, longchen@ust.hk  
https://github.com/HKUST-LongGroup/RECODE
Figure 1: Illustration of the challenges of VRD with similar relation categories holding and carrying. Four images and their ground-truths are on the left. The subject and object for each triplet are denoted by blue and pink boxes, respectively. (a) A child may incorrectly identify these two relations only based on similar concepts alone. (b) Using class-based prompts, CLIP always maps these two relations to adjacent locations in the semantic space. (c) We humans always utilize composite visual cues to correctly distinguish between different relations. (d) Our proposed RECODE uses LLM (e.g., GPT) to generate composite descriptions that aid the CLIP model in distinguishing between them.
Pretrained vision-language models, such as CLIP, have demonstrated strong generalization capabilities, making them promising tools in the realm of zero-shot visual recognition. Visual relation detection (VRD) is a typical task that identifies relationship (or interaction) types between object pairs within an image. However, naively utilizing CLIP with prevalent class-based prompts for zero-shot VRD has several weaknesses, e.g., it struggles to distinguish between different fine-grained relation types and it neglects essential spatial information of two objects. To this end, we propose a novel method for zero-shot VRD: RECODE, which solves RELATION detection via Composite DEscription prompts. Specifically, RECODE first decomposes each predicate category into subject, object, and spatial components. Then, it leverages large language models (LLMs) to generate description-based prompts (or visual cues) for each component. Different visual cues enhance the discriminability of similar relation categories from different perspectives, which significantly boosts performance in VRD. To dynamically fuse different cues, we further introduce a chain-of-thought method that prompts LLMs to generate rea
*Long Chen is the corresponding author. Work was done when Lin Li visited HKUST.
Figure 2: A comparative analysis of predictions made by RECODE and baseline CLIP using class-based prompts. It illustrates how our method offers interpretability to the relation classification results through the similarity  $\phi$  between the image and the description-based prompts.
sonable weights for different visual cues. Extensive experiments on four VRD benchmarks have demonstrated the effectiveness and interpretability of RECODE.
Recent advances in pretrained vision-language models (VLMs) [1, 2, 3, 4] (e.g., CLIP [1]), have shown remarkable generalization ability and achieved impressive performance on zero-shot recognition tasks. Specifically, CLIP employs two encoders: an image encoder that converts images into visual features, and a text encoder that transforms sentences into semantic features. This design allows the encoders to map different modalities into a common semantic space. When the inputs to the text encoder are class-based prompts, such as "A [CLASS]", "A photo of [CLASS]", CLIP can compare the image and prompts in the shared semantic space, thereby enabling zero-shot recognition of novel categories [1]. Compared to object recognition, visual relation detection (VRD) is much more challenging, which needs to identify the relation types between object pairs within an image in the form of \(\langle\) subject, relation, object \rangle\) [5, 6, 7, 8, 9]. It differs from object recognition in that it requires an understanding of how objects are related to each other. By crafting class-based prompts to describe these relation types, CLIP could potentially be extended to perform zero-shot VRD.
However, this straightforward baseline presents notable challenges. Imagine you are a child asked to distinguish relation categories "holding" and "carrying", both involving a person and an object. Based on the similar concepts of "holding" (i.e., a person having an object in their hands) and "carrying" (i.e., a person supporting an object in their hands), it would be difficult to determine the correct prediction (cf., Figure 1(a)). In other words, class-based prompts of "holding" and "carrying" might be projected to adjacent locations in semantic space by CLIP, leading to a relation sensitivity issue: CLIP struggles to differentiate between the subtle nuances of similar relations. Secondly, class-based prompts overlook the unique spatial cues inherent to each relation category, leading to a spatial discriminability issue. The "holding" category generally suggests the object being at a certain height and orientation relative to the person, while "carrying" implies a different spatial position, typically with the object located lower and possibly supported by the person's entire body. The neglect of spatial cues leads to inaccuracies in distinguishing between such spatial-aware relation categories. Moreover, applying CLIP in this manner brings about a computational efficiency issue. Using CLIP requires cropping each union region of a subject-object pair separately from the original image (i.e.,  $N^2$  crops for  $N$  proposals), leading to computational inefficiencies.
Nonetheless, we humans can distinguish relation categories from different visual cues. For example, from the subject's perspective, we could think that in the case of "holding", a person might be standing while having an object, such as an umbrella, in their hand. Meanwhile, in the case of "carrying", a person should be in a more engaged position, perhaps walking or running with both hands and arms supporting a heavy object, like a suitcase. In addition, spatial cues also play an important role in identifying these relation categories. For example, when a person is carrying an umbrella, the umbrella is usually positioned lower and closer to the person's body compared to when the person is holding an umbrella. Based on these visual cues, we can easily identify scenarios such as "person-holding-umbrella" and "person-carrying-umbrella" as in Figure 1(c).
Inspired by our humans' ability to extract and utilize different visual cues, we present a novel method for zero-shot VRD: RECODE, which classifies RELation via CO Composite DEscriptions. It first uses large language models (LLMs) [10], to generate detailed and informative descriptions $^2$  for different components of relation categories, such as subject, object, and spatial. These descriptions are then used as description-based prompts for the CLIP model, enabling it to focus on specific visual features that help distinguish between similar relation categories and improve VRD performance. Specifically, for the subject and object components, these prompts include visual cues such as appearance (e.g., with leg), size (e.g., small), and posture (e.g., in a sitting posture). For the spatial component, these prompts include cues related to the spatial relationships between objects, such as relative position and distance. By incorporating different visual cues, RECODE enhances the discriminability of similar relation categories, such as "riding" and "mounted" based on the different postures of the subject, e.g., "seated on the back of animal" for the subject of "riding". Similarly, spatial visual cues can be used to differentiate between "laying on" and "holding" based on the relative position between the subject and object, such as "subject above object" and "subject under object" (cf., Figure 2).
In addition, we explore the limitations of several description generation prompts for visual cue, e.g., relation class description prompt [11], and then design a guided relation component description prompt that utilizes the high-level object categories to generate more accurate visual cues for each relation category. For instance, if the high-level category of object is "animal", the generated object descriptions for relation "riding" are tailored to the "animal" category, e.g., "with four legs", instead of the "product", e.g., "with wheels". Meanwhile, to better fuse the evidence from different visual cues, we further leverage LLMs to predict reasonable weights for different components. Particularly, we design a chain-of-thought (CoT) method [12] to break down this weight assignment problem into smaller, more manageable pieces, and prompt LLM to generate a series of rationales and weights.
To evaluate our RECODE, we conducted experiments on four benchmark datasets: Visual Genome (VG) [13] and GQA [14] datasets for scene graph generation (SGG), and HICO-DET [15] and V-COCO [16] datasets for human-object interaction (HOI) detection. Experimental results prove the generalization and interpretability of our method. In summary, we made three main contributions in this paper: 1) We analyze the weaknesses of the prevalent class-based prompt for zero-shot VRD in detail and propose a novel solution RECODE. RECODE leverages the power of LLMs to generate description-based prompts (visual cues) for each component of the relation class, enhancing the CLIP model's ability to distinguish between various relation categories. 2) We introduce a chain-of-thought method that breaks down the problem into smaller, more manageable pieces, allowing the LLM to generate a series of rationales for each cue, ultimately leading to reasonable weights for each component. 3) We conduct experiments on four benchmark datasets and demonstrate the effectiveness and interpretability of our method.
Typically, VRD is comprised of two sub-tasks: object detection and relation classification [5]. Since zero-shot object detection has been extensively studied [17, 1, 11], in this paper, we primarily focus on zero-shot relation classification. Specifically, given the bounding boxes (bboxes)  $\{b_i\}$  and object categories  $\{o_i\}$  of all objects, our target is to predict the visual relation (or predicate/interaction) categories  $\{r_{ij}\}$  between pairwise objects. To facilitate presentation, we use  $s$ ,  $o$ , and  $p$  to denote the subject, object, and their spatial position in a triplet respectively, and  $r$  to denote the relation category.
Class-based Prompt Baseline for Zero-Shot VRD. Following recent zero-shot object recognition methods, a straightforward solution for zero-shot VRD is the CLIP with class-based prompt. Specif-
2We use a description to represent a visual cue.
Figure 3: The framework of RECODE. 1) Visual feature decomposing decomposes the triplet into subject, object, and spatial features. 2) Semantic feature decomposing decomposes relation categories into subject, object, and spatial descriptions. 3) Relation classification calculates similarities between decomposed visual and semantic features and applies softmax to obtain the probability distribution.
ically, a pretrained CLIP image encoder  $V(\cdot)$  and a pretrained CLIP text encoder  $T(\cdot)$  are used to classify pairwise objects with a set of relation classes. For each relation class, a natural language class-based prompt  $p_c$  is generated, incorporating the relation information, e.g., "[REL-CLS]-ing/ed" or "a photo of [REL-CLS]". Each prompt is then passed through  $T(\cdot)$  to get semantic embedding  $t$ , while the union region of a subject-object pair is passed through  $V(\cdot)$  to get visual embedding  $v$ . The cosine similarity between  $v$  and  $t$  of different relation categories is calculated and processed by a softmax function to obtain the probability distribution over all relation categories.
2.1 Zero-shot VRD with Composed Visual Cues
To overcome the limitations of class-based prompts, we propose a novel approach RECODE for zero-shot VRD. It consists of three parts: visual feature decomposing, semantic feature decomposing, and relation classification (cf., Figure 3). In the first two parts, we decompose the visual features of the triplet into subject, object, and spatial features, and then generate semantic features for each component. In the last part, we calculate the similarities between the decomposed visual features and a set of semantic features, and aggregate them to get the final predictions over all relations.
Visual Feature Decomposing. To enhance spatial discriminability and computational efficiency, we decompose the visual features of a triplet into subject, object, and spatial features. For subject and object features, we crop the regions of the subject and object from the original image using the given bboxes  $b_{s}$  and  $b_{o}$ , and encode them into visual embeddings  $v_{s}$  and  $v_{o}$  using the image encoder  $V(\cdot)$  of CLIP. For spatial features, we aim to obtain the spatial relationship between the subject and object based on their bounding boxes. However, directly obtaining all spatial images based on the given bounding boxes is computationally expensive due to the diversity of spatial positions ( $N^2$  each image). To address this, we simulate the spatial relationship between the subject and object using a finite set of spatial images, represented by red and green bboxes respectively. We define four attributes (shape, size, relative position, and distance) based on bounding box properties. Each attribute is assigned a finite set of values to construct a finite set of simulated spatial images. For a given triplet, we match the calculated attribute values with the most similar simulated image<sup>3</sup>. The matched spatial image is then encoded into a visual embedding  $v_{p}$  using  $V(\cdot)$  of CLIP.
Semantic Feature Decomposing. To improve the CLIP model's ability to distinguish between different relation classes, we incorporate a set of description-based prompts  $D$  to augment the original class-based prompt for each relation category. For the subject and object components, we generate a set of description-based prompts  $D_{s}$  and  $D_{o}$  to provide additional visual cue information, the generation process is described in Sec. 2.2. These prompts contain object categories with specific visual cues that highlight the unique characteristics of the relation being performed, e.g., "women, with legs", which enhances the discriminability between similar relation categories. For the spatial component, it only contains a set of description-based prompts  $D_{p}$  that include information about the relative position and distance between the subject and object in the image. By incorporating
<sup>3</sup>Due to the limited space, the details are left in the appendix.
Figure 4: Examples of different prompts used for generating descriptions of visual cues. (a) Relation class description generates descriptions for each relation class directly. (b) Relation component description generates descriptions for each component of the relation separately. (c) Guided relation component description incorporates high-level object category guide generation process.
this additional information, we aim to distinguish between relations based on spatial location. After generating these sets of description-based prompts, we obtain semantic embeddings  $\{t_{d_{s_i}}\}, \{t_{d_{o_i}}\}$ , and  $\{t_{d_{p_i}}\}$  using a text encoder  $T(\cdot)$ , separately. These embeddings, along with the class-based prompt embedding  $t_c$ , are used for relation classification.
Relation Classification. In this step, we compute the similarity score between the visual and semantic features to obtain the relation probability distribution. We first calculate the cosine similarity  $\phi(\cdot, \cdot)$  between each visual embedding and semantic embedding for each relation category  $r$ . The final score incorporates both class-based and description-based prompts, and is calculated as follows:
$$
S (r) = \underbrace {\phi \left(v _ {s} , t _ {c}\right) + \phi \left(v _ {o} , t _ {c}\right)} _ {\text {c l a s s - b a s e d p r o m p t s}} + \underbrace {\sum_ {k \in \{s , o , p \}} \frac {w _ {k}}{\left| D _ {k} (r) \right|} \left[ \sum_ {d _ {k _ {i}} \in D _ {k} (r)} \phi \left(v _ {k} , t _ {d _ {k _ {i}}}\right) \right]} _ {\text {d e s c r i p t i o n - b a s e d p r o m p t s}}, \tag {1}
$$
where  $w_{k}$  represents the importance of visual cues for each component  $k \in \{s, o, p\}$ , and  $|D_{k}(r)|$  denotes the number of visual cues in  $D_{k}(r)$  for relation category  $r$ . We compute the similarity of individual visual cues for each component and then obtain their average. The weights of different components are determined by a LLM, which will be discussed in Sec. 2.2. Finally, we apply a softmax operation to the scores to obtain the probability distribution over all relation categories.
2.2 Visual Cue Descriptions and Weights Generation
LLMs, such as GPT [10], have been shown to contain significant world knowledge. In this section, we present the process of generating descriptions of visual cue  $D_{s}$ ,  $D_{o}$ , and  $D_{p}$ , as well as the weights  $w_{s}$ ,  $w_{o}$ , and  $w_{p}$  for each component of each relation category using LLMs.
2.2.1 Visual Cue Descriptions
In this section, we explore methods for generating descriptions of visual cues for relation decomposition. Inspired by the work [11] of zero-shot image classification, we first propose relation class description prompt, which generates descriptions from the perspective of class-level (cf., Figure 4(a)). It has the advantage of producing descriptions that are easy to interpret and understand. However, it may result in overly diverse and information-rich descriptions that could hinder the extraction of meaningful visual cues, e.g., "speed of the person" in Figure 4(a).
To address this limitation, we then consider another relation component description prompt, which involves decomposing the relation into its subject and object components and generating descriptions of their visual features separately (cf., Figure 4(b)). While this type of prompt allows for more focused and specific descriptions of visual cues, it may not be effective in capturing the variations in visual features between different subject-object category pairs. For example, "man-riding-horse" and "person-riding-bike" typically have totally different visual features for the object. The visual cues "reins" and "saddle" of the object in Figure 4(b) are inappropriate for a "bike".
(a) Weight Prompt
Suppose you are a relation classification model.  
Given: subject belongs to "human" and object belongs to "product".
The visual features of subject: with eyes directed towards the object, with head upright.
The visual features of object: with visible features such as front, display, or screen.
The visual features of position: subject positioned either above, below, left or right of the object at a mid distance. Q: How do you weight these visual features to determine the relation is "looking at"? The sum of weights must be 1.0!
Figure 5: Illustration of the effectiveness of CoT method in generating reasonable visual cue weights. (a) Prompt without CoT. LLM assigns same weights for subject and object. (b) Prompt with CoT. LLM analyzes the importance of each cue step by step and assigns more reasonable weights.
Visual features of subject: 0.4
Visual features of object: 0.4
Visual features of position: 0.2
(b) Weight Prompt with Chain-of-Thought
The sum of weights must be 1.0!
A: Let's think step by step! (CoT prompt)
First, we need to consider the importance of the subject's visual features. Since the direction of the eyes and head position strongly indicate the focus of attention, we will give them a weight of 0.6.
Next, we need to consider the importance of the object's visual features. Since the visible features such as front, display, or screen indicate that the object is something that can be looked at, we will give them a weight of 0.3.
Finally,..., we will give them a weight of 0.1.
Therefore, we can weight these visual features as follows:
Visual features of subject: 0.6
Visual features of object: 0.3
Visual features of position: 0.1
Therefore, we design guided relation component description prompt. It builds upon the second method by incorporating the high-level category information of the object into the generation process, leading to more accurate and informative descriptions of the visual features of both the subject and object components (cf., Figure 4(c)). To achieve this, we classify the object into high-level classes, such as "human", "animal", and "product", to guide the description generation. For example, "bike" is classified as "product", and "horse" is classified as "animal". This allows for the separate generation of visual feature descriptions for each high-level object class, e.g., "a harness or saddle on its body" for "animal", resulting in more precise and relevant visual cues for each relation category<sup>3</sup>.
2.2.2 Visual Cue Weights
Intuitively, different combinations of visual cues may have varying degrees of importance in relation classification. For example, for relation "looking at", the visual cue "with visible features" of the object may not be as informative as the visual cue "with eye" of the subject. To account for this, we leverage the impressive knowledge and reasoning abilities of LLMs to analyze the discriminative power of different visual cues and dynamically assign weights accordingly. Specifically, we provide each combination of visual cues as input to LLM and prompt it to determine the appropriate weight for each cue for distinguishing the given predicate. The prompts used for this purpose are in Figure 5.
Chain-of-Thought (CoT) Prompting. To ensure the generated weights are reasonable, we utilize a CoT method that has demonstrated remarkable reasoning abilities [12, 18]. Specifically, we prompt the LLM to generate rationales by using the stepwise reasoning prompt "Let's think step by step!" to break down the problem into smaller, more manageable pieces. Then LLM generates a series of rationales, and those that lead to the reasonable weights. For example in Figure 5, we demonstrate the importance of the CoT method in generating more accurate weights. Without the stepwise reasoning prompt, LLM generates the same weight for both the subject and object visual cues for "looking at", which is clearly unreasonable. However, with the CoT prompt, LLM is able to analyze each cue step by step, leading to a more accurate assignment of weights, i.e., the cues about the subject are relatively more important. In order to standardize the format of the strings generated by LLMs for extracting different components of visual cues and weights, we make certain modifications to the prompts for descriptions and weights<sup>3</sup>.
Datasets. We evaluated our method on four zero-shot VRD benchmarks: 1) VG [13] contains 26,443 images for testing, each annotated with object and predicate labels to form a scene graph. Following previous works [5], we used the pre-processed VG with 150 object classes. We adopted the 24 semantic predicate classes proposed in [19, 20], as they are more informative and challenging for classifying. 2) GQA [14] is a large-scale SGG dataset. We used the same split provided by [21], which contains 8,208 images for testing with 200 object classes. As for predicate classes, we selected
Table 1: Evaluation results on the test set of VG and GQA datasets. † denotes removing the guidance from high-level object category. ☆ denotes integrated with Filter strategy.
Data	Method	Predicate Classification
R@20	\( \bigtriangleup \)	R@50	\( \bigtriangleup \)	R@100	\( \bigtriangleup \)	mR@20	\( \bigtriangleup \)	mR@50	\( \bigtriangleup \)	mR@100	\( \bigtriangleup \)
VG	CLS	7.2	-	10.9	-	13.2	-	9.4	-	14.0	-	17.6	-
CLSDE	7.0	-0.2	10.6	-0.3	12.9	-0.3	8.5	-0.9	13.6	-0.4	16.9	-0.7
RECODE†	7.3	0.1	11.2	0.3	15.4	2.2	8.2	-1.2	13.5	-0.5	18.3	0.7
RECODE	9.7	2.5	14.9	4.0	19.3	6.1	10.2	0.8	16.4	2.4	22.7	5.1
RECODE*	10.6	3.4	18.3	7.4	25.0	11.8	10.7	1.3	18.7	4.7	27.8	10.2
GQA	CLS	5.6	-	7.7	-	9.9	-	6.3	-	9.5	-	12.2	-
CLSDE	5.4	-0.2	7.2	-0.5	9.3	-0.6	6.0	-0.3	8.8	-0.7	11.5	-0.7
RECODE†	5.2	-0.4	7.8	0.1	10.2	0.3	5.8	-0.5	8.9	-0.6	11.3	-0.9
RECODE	6.3	0.7	9.4	1.7	11.8	1.9	7.8	1.5	11.9	2.4	15.1	2.9
RECODE*	7.0	1.4	11.1	3.4	15.4	5.5	9.4	3.1	14.8	5.3	20.4	8.2