SlowGuess/NeurIPS_2023_OCR huggingface dataset:
26 semantic predicate classes by referring to VG. 3) HICO-DET [15] contains 9,658 testing images annotated with 600 HOI triplets derived from combinations of 117 verb classes and 80 object classes. 4) V-COCO [16] comprises 4,946 testing images annotated with 29 action categories.
Evaluation Metrics. For SGG datasets (i.e., VG and GQA), we reported Recall@K ( $\mathbf{R}@\mathbf{K}$ ) which indicates the proportion of ground-truths that appear among the top-K confident predictions, and mean Recall@K ( $\mathbf{mR}@\mathbf{K}$ ) which averages  $\mathbf{R}@\mathbf{K}$  scores calculated for each category separately [22]. For HOI datasets (i.e., HOI-DET and V-COCO), we reported mean Average Precision ( $\mathbf{mAP}$ ) [23].
Implementation Details. For the LLM, we employed the GPT-3.5-turbo, a highly performant variant of the GPT model. As for CLIP, we leveraged the OpenAI's publicly accessible resources, specifically opting for the Vision Transformer with a base configuration (ViT-B/32) as default backbone<sup>3</sup>.
Settings. The bounding box and category of objects were given in all experiments. We compared our RECODE with two baselines: 1) CLS, which uses relation-CLasS-based prompts (e.g., "riding") to compute the similarity between the image and text. 2) CLSDE, which uses prompts of relation CLasS DEscription as shown in Figure 4(a). Each component of the proposed framework can serve as a plug-and-play module for zero-shot VRD. Specifically, 1) Filter, which denotes filtering those unreasonable predictions (e.g., kid-eating-house) with the rules generated by  $\mathrm{GPT}^3$ . 2) Cue, which denotes using description-based prompts (Sec. 2.1). 3) Spatial, which denotes using spacial images as additional features. 4) Weight, which denotes using dynamic weights generated by GPT to determine the importance of each feature, i.e., visual cue weights.
In this work, we evaluated the prediction performance of the proposed framework on two related tasks, i.e., SGG and HOI. The former outputs a list of relation triplet  $\langle \text{sub}, \text{pred}, \text{obj} \rangle$ , while the latter just fix the category of sub to human. Overall, our method achieved significant improvement on the two tasks compared to the CLS baseline, which shows the superiority of our method.
Evaluation on SGG. From the results in Table 1, we have the following observations: 1) CLSDE showed worse performance than the trivial CLS baseline. This is because the considerable noise in CLSDE which may hinder the model to attend the most distinguishable parts. 2) With the proper guidance, RECODE achieved considerable improvements compared to the baselines, e.g.,  $0.8\%$  to  $6.1\%$  gains on VG and  $0.7\%$  to  $2.9\%$  gains on GQA. The
Table 2: Evaluation results on the test set of HICO-DET and V-COCO datasets.
Method	HICO-DET	V-COCO
Full	Rare	Non-Rare	Role 1	Role 2
CLS	32.3	33.2	31.8	25.5	28.6
CLSDE	32.5	33.1	32.2	25.6	28.8
RECODE†	32.5	33.0	32.4	25.7	28.8
RECODE	32.7	33.2	32.5	26.0	29.0
performance drops of RECODE† also demonstrated the importance of guidance from high-level object categories during the generation process. 3) Integrated with the filtering strategy, i.e., RECODE*, achieved the best performance over all metrics, which suggests that commonsense knowledge is complementary and effective for zero-shot VRD. It also demonstrated that CLIP struggles to distinguish abstract concepts, i.e., relation sensitivity as mentioned in Sec. 1.
Table 3: Ablation studies on different architectures of CLIP. The official released weights are used.
Architecture	Method	Predicate Classification
R@20	R@50	R@100	mR@20	mR@50	mR@100
ViT-L/14	CLS*	8.3	15.0	21.5	7.6	14.2	24.2
RECODE*	11.2	19.9	28.0	9.1	18.5	28.1
ViT-L/14@336px	CLS*	8.6	15.4	21.8	7.7	13.9	23.0
RECODE*	12.1	21.1	29.2	9.7	19.5	28.2
ViT-B/32	CLS*	7.5	13.7	19.4	9.1	15.9	24.0
RECODE*	10.6	18.3	25.0	10.7	18.7	27.8
ViT-B/16	CLS*	8.6	15.5	22.1	9.8	17.2	25.2
RECODE*	12.6	21.0	28.5	12.5	20.2	30.0
Evaluation on HOI. Since standard evaluation procedure of HOI had filtered out those unreasonable predictions, RECODE* was not evaluated here. From the results in Table 2, we can observe that the performance gains were lower than those on SGG, e.g.,  $0.0\%$  to  $0.7\%$  gains on HICO-DET and  $0.4\%$  to  $0.5\%$  gains on V-COCO. The reasons are two-fold. On the one hand, since the category of subject is always a human, its features are too similar to be distinguished by CLIP. On the other hand, some of the actions are very similar in appearance. For example, distinguishing between actions like "person-throw-sports ball" and "person-catch-sports ball" is challenging due to their visual similarity.
3.3 Diagnostic Experiment
Architectures. We investigated the impact of changing the architectures of CLIP, as shown in Table 3. From the results, we can observe consistent improvements regardless of the architecture used.
Key Component Analysis. The results are summarized in Table 4. The first row refers to the CLS baseline. Four crucial conclusions can be drawn. First, with the guidance of Cue, consistent improvements can be observed, e.g.,  $0.2\%$  to  $3.4\%$  gains on  $\mathrm{R@K}$  w/o Filter and  $1.3\%$  to  $4.1\%$  gains on  $\mathrm{R@K}$  with Filter. Second, by introducing the spatial feature, the relative position of subject and object is considered, resulting in notable perfor
Table 4: Analysis of key components on the test set of VG.
Filter	Cue	Spatial	Weight	Predicate Classification
R@20	R@50	R@100	mR@20	mR@50	mR@100
7.2	10.9	13.2	9.4	14.0	17.6
✓	7.4	12.3	16.6	9.0	14.0	19.5
✓	✓	9.1	13.4	17.4	9.3	15.0	20.3
✓	✓	7.9	13.4	17.7	9.3	14.7	20.5
✓	✓	✓	9.7	14.9	19.3	10.2	16.4	22.7
✓	7.5	13.7	19.4	9.1	15.9	24.0
✓	✓	8.8	15.9	23.5	10.3	17.2	26.2
✓	✓	✓	9.3	16.3	22.5	10.1	18.1	25.5
✓	✓	✓	10.0	17.5	24.8	10.4	17.8	26.7
✓	✓	✓	✓	10.6	18.3	25.0	10.7	18.7	27.8
mance gains on R@K (0.8% to 1.7%) and mR@K (0.3% to 1.0%) w/o Filter compared to just using Cue. This is because the spatial feature is of importance for relation detection [6]. Third, benefiting from the impressive reasoning ability of LLMs, the proposed weighting strategy can determine the importance of different cues, thus achieving further improvements, e.g., 0.5% to 1.1% gains on R@K compared to average aggregation. Fourth, by filtering those unreasonable predictions, consistent improvements can be observed. The reason may be that the performance of relation detection of CLIP is not accurate enough. Empirically, commonsense knowledge is a feasible way to filter those noise. Combining all components allows for the best overall performance on all evaluation metrics.
Case study. To investigate the most important regions for distinguishing relations, we visualized the attention map given different images and prompts (cf., Figure 6). From the visualization of class-based prompts, we can observe that CLIP may attend those regions unrelated to the query prompts, e.g., focusing on the body of a person given relation "growing on". We attribute this phenomenon to the insufficient information within given prompts, which is also our motivation to introduce visual cue descriptions. As for description-based prompts, CLIP can attend to right regions with the guidance of descriptions, e.g., focusing on colorful patterns on the product given relation "painted on".
Figure 6: Visualization of CLIP attention maps on input images with different prompts. The right side shows the partial description-prompts generated for each predicate category given the high-level object category. They are used to generate the corresponding attention maps on the right.
Visual Relation Detection (VRD) aims to predict the relationships of given subject-object pairs, which can be viewed as a pair-wise classification task and have been widely studied in the image domain, e.g., scene graph generation (SGG) [5, 6, 22, 24] and human-object interaction (HOI) detection [25, 26, 27]. Previous solutions mainly focus on learning representations from the training samples on pre-defined categories, which may suffer from noisy annotations [22] or long-tailed predicate distribution [6, 28] and are far from the needs of the real-world scenarios. Recently, some attempts [8, 29] adopted prompt-tuning [30] to predict unseen categories during inference. However, since the learnable prompts may be overfitting when trained on seen categories, their performance is sensitive to the split of seen/unseen categories [31]. In contrast, our method can predict the relationships directly without any training samples, and has better interpretability and generalization ability, especially in rare informative relation categories.
Zero-shot Visual Recognition enables the model to recognize new categories that it has never seen during training, which is one of the research hotspots in the vision community. Aligning visual representations to pre-trained word embeddings (e.g., Word2Vec [32] and GloVe [33]) is an intuitive and feasible way to achieve this goal [34]. More recently, VLMs, which use contrastive learning [35] to learn a joint space for vision and language, have demonstrated their impressive zero-shot ability [1]. Therefore, many zero-shot works [36, 37, 38] adopted such VLMs as their basic component to use the knowledge of the learned joint space. However, most of them only utilized the class name of unseen categories during inference, which makes an over-strong assumption that the text encoder project proper embeddings with only category names [11]. Then, Menon and Vondrick [11] proposed to query LLMs for the rich context of additional information. Nonetheless, it is non-trivial to apply such paradigms to VRD as discussed in Sec. 1. To the best of our knowledge, we are the first to leverage both LLMs and VLMs for VRD in an efficient, effective, and explainable way.
In this paper, we proposed a novel approach for zero-shot Visual Relationship Detection (VRD) that leverages large language models (LLMs) to generate detailed and informative descriptions of visual cues for each relation category. The proposed method addresses the limitations of traditional class-based prompts and enhances the discriminability of similar relation categories by incorporating specific visual cues. Moreover, we introduced a chain-of-thought method that breaks down the problem into smaller, more manageable pieces, allowing the LLM to generate a series of rationales for each visual cue and ultimately leading to reasonable weights. Our experiments on four benchmark datasets demonstrated the effectiveness and interpretability of our method.
Acknowledgement. This work was supported by the National Key Research & Development Project of China (2021ZD0110700), the National Natural Science Foundation of China (U19B2043, 61976185), and the Fundamental Research Funds for the Central Universities (226-2023-00048). Long Chen is supported by HKUST Special Support for Young Faculty (F0927), and HKUST Sports Science and Technology Research Grant (SSTRG24EG04).
[1] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748-8763, 2021.
[2] Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannaneh Hajishirzi, Ali Farhadi, Hongseok Namkoong, et al. Robust fine-tuning of zero-shot models. In CVPR, pages 7959-7971, 2022.
[3] Jaemin Cho, Seunghyun Yoon, Ajinkya Kale, Franck Dernoncourt, Trung Bui, and Mohit Bansal. Fine-grained image captioning with clip reward. In Findings of NAACL, 2022.
[4] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, pages 4904-4916, 2021.
[5] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, pages 5410–5419, 2017.
[6] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In CVPR, pages 3716-3725, 2020.
[7] Xingchen Li, Long Chen, Jian Shao, Shaoning Xiao, Songyang Zhang, and Jun Xiao. Rethinking the evaluation of unbiased scene graph generation. In BMVC, 2022.
[8] Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Towards open-vocabulary scene graph generation with prompt-based finetuning. In ECCV, 2022.
[9] Lin Li, Guikun Chen, Jun Xiao, Yi Yang, Chunping Wang, and Long Chen. Compositional feature augmentation for unbiased scene graph generation. In ICCV, pages 21685-21695, 2023.
[10] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NIPS, 2020.
[11] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2022.
[12] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NIPS, 2022.
[13] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
[34] Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In CVPR, pages 6857-6866, 2018.
[35] Ching-Yao Chuang, Joshua Robinson, Yen-Chen Lin, Antonio Torralba, and Stefanie Jegelka. Debiased contrastive learning. In NIPS, 2020.
[36] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In CVPR, pages 14393-14402, 2021.
[37] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR, 2022.
[38] Tao Wang and Nan Li. Learning to detect and segment for open vocabulary object detection. In CVPR, 2023.