SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 2: The Venn diagram of knowledge comparison between the open visual knowledge from OpenVik with the non-parametric knowledge from existing knowledge graph (i.e., ConceptNet) and parametric knowledge from large language model (i.e., COMET).
Knowledge Source	Examples
Non-parametric Knowledge	dog\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\uunderline
dog\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\u\uunderline
We compare the extracted visual knowledge with the non-parametric knowledge in the existing knowledge graph (KG) and the parametric knowledge from the large language model (LLM). The comparison insights from the three knowledge resources are shown in the Venn Diagram in Figure 2.
Compare with non-parametric knowledge. We take ConceptNet [41] as the representative in the comparison with non-parametric knowledge. To map the knowledge generated by OpenVik to ConceptNet, we parse the knowledge into triplets and associate the endpoints of these triplets with
nodes in ConceptNet. Then we calculate the similarity of embeddings $^3$  between the parsed relation and all the edge relations among the mapped nodes in ConceptNet. If the similarity score exceeds a predetermined threshold, i.e., 0.75, we consider the mapping successful. As illustrated in Figure 2, we observe that compared with the non-parametric knowledge in KG, the extracted visual knowledge captures richer and more meaningful spatial details, e.g., "three layer cake on table", and motion dynamics, e.g., "baby elephants walking around adventurous wood".
Compare with parametric knowledge. We compare with parametric knowledge contained in LLM by prompting the gpt-3.5-turbo $^4$  model with the object information in the image. The prompt template used is detailed in Appendix E. The mapping process follows the approach mentioned earlier. It is found that compared with the parametric knowledge in LLM, the extracted visual knowledge exhibits unique fine-grained visual details, e.g., "red sticker on fence", and provides precise scene information, e.g., "the light shining from bright black background".
The influence on knowledge quality with information variety regularization and data strategies. We conducted ablation studies to evaluate the effectiveness of the information variety regularizer,  $\mathcal{L}_{\mathrm{V}}$  and our diversity-driven data enhancement strategies. This involves an in-depth assessment of knowledge quality on the same eval
Figure 3: The influence of information variety regularization and diversity-driven data enhancement strategies.
valuation subset. The results are presented in Figure 3. It is evident from the results that our proposed information variety design primarily impacts freshness and diversity, without compromising validity and conformity. For the freshness, the omission of data augmentation for entities and relations results in the most significant performance degradation. This implies the crucial role these strategies play in infusing novel knowledge into the generation process. As for diversity, the most notable changes in metrics are observed when the  $\mathcal{L}_{\mathrm{V}}$  and random dropping are removed. The strategy for augmenting entities and relations also plays a valuable role in enriching diversity.
Ablation of the pre-training for the open relational region detector. We conducted a comparison of the outcomes when loading a pre-trained detector backbone versus training the detector from scratch, as shown by the yellow bar in Figure 3. Results demonstrate a noticeable decrease in both knowledge diversity and freshness, which indicates the importance of loading the pre-trained model for region detection. This may be because omitting the pre-training step of the FasterRCNN model tends to result in the detection of more overlapping regions, which in turn causes the drop.
The influence on dataset diversity with data strategies. We conduct a direct analysis of the knowledge diversity of the existing datasets and our diversity-enhanced one, compared with the visual knowledge generated from OpenVik. The findings, presented in Table 4, show that the diversity-driven data enhancement strategies significantly boost knowledge diversity. Trained with this enhanced data, OpenVik can extract visual knowledge that exhibits greater diversity than that found in the Visual Genome and Relational Caps, indicating the advantage of OpenVik to format-free visual knowledge generation and its ability to yield richer knowledge diversity.
Table 4: Diversity of existing and enhanced datasets and generated knowledge from OpenVik.
Metrics	Training Dataset	Generate Knowledge
Visual Genome [24]	Relational Caps [22]	Diversity Enhanced (Ours)	OpenVik (Ours)
Diversity	0.589	0.604	0.632	0.619
We present two case studies in Figure 4 (See Appendix F for more) to showcase the format-free visual knowledge generated by OpenViK, in comparison to Visual Genome (Scene Graph and Region Description) and Relational Caps. Contrary to the rigidity of scene graphs, which strictly adhere to a
Embeddings are produced by ConceptNet API: https://github.com/commonsense/conceptnet-numberbatch.
4https://platform.openai.com/docs/models/gpt-3-5
predefined format, OpenVik can generate knowledge with a flexible semantic structure, not strictly bound to the sub-verb-object format (e.g., "blue post attached to wall with white letter"). Examples of this adaptability are highlighted in red. When compared to dense region descriptions, the relational knowledge extracted by OpenVik offers a deeper understanding of the multiple entity interactions within an image. In comparison to Relational Caps, which mainly focus on interactions between two objects, OpenVik significantly broadens the diversity of relation with vivid verbs (e.g., "attached to", "adorning"). Moreover, it introduces novel entities (e.g., "post", "mane") and enriches the knowledge representation with nuanced details (e.g., "full of", "striped") that are missed by Relational Caps.
Visual Genome-Scene Graph:
<drink, in, cooler>
<orange, in, box>
<banner, on, building>
<item, on, table>
Visual Genome-Region Descriptions: oranges in a wood thing green leaves on oranges red writing on a white sign drink in red cooler
Visual Genome-Scene Graph:
Visual Genome-Region Descriptions:
<hair, on, head>
<zebra, eat, grass>
<eye, on, zebra>
<grass, on, ground>
black and white striped leg
light shining on the zebra
thin line of black hair
two zebras grazing in the grass
OpenVik:
blue post attached to wall with white letter
the open window to snowy ground
- wood box full of different size of orange
□ white banner on a building with letter o
blue box sitting beside a sneaky garage
a orange covered with ice and green leaves
Relational Caps:
snow-covered oranges in wood thing the frost on snow-covered oranges green leaves on snow-covered oranges red writing on white sign
striped mane belongs to grazing zebra
zebra with striped ears eating green grass
white stripe adorning leg
dark brown mane growing behind head
□ grass everywhere surround standing zebra
black nose above green lively grass
sticking up ear of grazing zebra
black eye of eating zebra
grazing zebra in green grass
the muzzle of grazing zebra
Figure 4: Case study on the extracted open visual knowledge from OpenVik. Examples of format-free knowledge are highlighted in red. Compared with VG and Relational Caps, OpenVik performs better at capturing novel entities, broadening object interactions with diverse relations, and enriching the knowledge representation with nuanced descriptive details.
Note that we observe the unbalanced and noisy distributions within the training data can lead to errors in the knowledge produced. Viewing hallucinations as erroneous inferences based on input, the inaccuracies observed in OpenVik and similar baselines often stem from detection errors. These errors are typically caused by data biases that incorrectly associate features with a specific class or label. We further two illustrative failure cases in Figure 5. For example, a "black speaker by flat tv" is generated, although the speaker is not present in the image—possibly reflecting common co-occurrences within the dataset. Similarly, a ladder in the right figure has been misidentified as a towel, leading to the erroneous description of a "blue towel hanging from dry shower". The key to mitigating such incorrect inference is identifying the cofounder feature of class labeling.
<window, in, screen>
<eye, of, cat>
<keyboard, on, laptop>
<cat, beside, laptop>
Visual Genome-Region Descriptions:
laptop key board
cat beside the laptop
pictures on wall behind cat
the cat is brown
white cat beside black laptop
the window in screen
the wall behind cat
<paint, in, frame>
<picture, on, wall>
<window, in, bathroom>
<curtain,overwindow>
leg of an elephant
baby elephant in grass
Grass within the enclosure
A big rock on the ground
framed picture hanging on wall
recessed lighting in ceiling
blue towel hanging from dry shower
the sunlight coming through window
blue painting in metal frame
the shampoo bottle on rack
blue jar in window
white cat sitting near computer
black speaker by flat tv
black wire plugged into computer
the picture in screen
Figure 5: Examples of incorrectly knowledge resulting from distribution bias are highlighted.
5 Application
This section explores whether the extracted open visual knowledge from OpenVik can bolster reasoning and inference capabilities in multimodality downstream tasks by augmenting a baseline in the challenging zero-shot setting.
5.1 Text-to-Image Retrieval
Task Setting. In the text-to-image retrieval task, a given caption is matched to a large set of candidate images, with the most relevant image returned as the result. Adopting the challenging zero-shot
setting, we generate the visual representation  $\mathbf{v}$  and textual representation  $t$  of the given image  $\mathcal{T}$  and caption  $\mathcal{T}$  using a pre-trained clip-retrieval model [3]. The baseline involves the image and text embedding similarly based on zero-shot CLIP Retrieval [3] and the fine-tuned model from BLIP [26].
To explore the potential of the extracted visual knowledge from OpenViK, we enrich the given caption  $\mathcal{T}$  with related contexts derived from the extracted visual knowledge. Specifically, for each query caption, we parse the caption to extract all subject-object pairs  $(s,o)$  with the NLTK parser. Then  $s$  and  $o$  are mapped to the open visual knowledge, where knowledge phrases that contain relations occurring more than  $30\%$  of the time between  $s$  and  $o$  are enriched to the original caption  $\mathcal{T}$ .
Figure 6: An example of OpenVik enrichment on text-to-image retrieval (See Appendix G.1 for more).
Original text: A little room and dining room area with furniture. A living room with a big table next to a book shelf. A living room decorated with a modern theme. A living room with wooden floors and furniture. The large room has a wooden table with chairs and a couch.
Enriched text: big table in room. a decorated living room with wooden furniture. brown couch in room. book on table. wooden table with shelf. shelf next to couch. wooden bookshelf with books next to table.
Method	Recall@1	Recall@5	Recall@10	\(Avg\)
ZS-CLIP	36.16	65.47	78.66	60.10
OpenVik + ZS-CLIP	40.55	73.29	84.53	66.12
BLIP	63.11	86.30	91.10	80.17
OpenVik + BLIP	65.23	87.71	91.90	81.61
Table 5: Text-to-image retrieval results (%) of OpenVik enrichment compared with zero-shot baselines.
Qualitative examples. Figure 6 presents an example of OpenVik-based visual knowledge enrichment on captions. By incorporating related contexts from the generated open visual knowledge, the enriched captions convey more precise visual details, which enhances the alignment for text-image alignment.
Quantitative results. We curated a subset of 680 images from the testing set of the MS-COCO dataset containing parsed knowledge with at least eight nouns. This ensures an adequate degree of enrichment is achieved through the use of OpenVik. Standard image retrieval metrics, i.e., Recall@1/5/10/ and Avg, are employed to evaluate the performance. The results are presented in Table 5. It is evident that relational context enrichment leads to the average correction of more than  $6.0\%$  of the initial zero-shot, highlighting the practical benefits of extracted visual knowledge in visual reasoning tasks.
5.2 Grounded Situation Recognition
Task setting. The event type prediction for the grounded situation recognition task is to predict the best match from predefined 504 event types [36] based on the image. We convert each candidate event verb into a description  $\mathcal{T}$ : "An image of  $<\text{verb}>$ " for image description matching. Similarly to text-to-image retrieval, we include zero-shot CLIP and the fine-tuned model from BLIP as baselines.
To enrich with contextual knowledge from OpenVik, for each given verb  $v$ , we find its nearest synonym in the extracted open visual knowledge and enrich the text description with the most common knowledge phrase containing it, regularized by the objects present in the image. Instead of directly concatenating the retrieved knowledge triplets to the original textual description, we employ an additive decomposition strategy: the similarity  $s(\mathcal{I}, v)$  of the candidate verb  $v$  with respect to the given image  $\mathcal{I}$  is calculated as  $s(\mathcal{I}, v) = \frac{1}{|D(v)|} \sum_{d \in D(v)} \phi(\mathcal{I}, v)$ , where  $D(v)$  is the set of descriptors, including the original description and the enriched ones, and  $\phi$  represents the single log probability that descriptor  $d$  pertains to the image  $\mathcal{I}$ .
Verb: talking
Figure 7: An example of OpenVik context enrichment on task GSR (See Appendix G.2 for more).
Original text: This is an image of talking. Enriched text: man talking on a small white telephone. adult male with white shirt talking on chatty cellphone. the man waving arms is talking on phone.
Method	Accuracy	Precision	Recall	F1
ZS-CLIP	53.14	42.54	45.19	43.82
OpenVik + ZS-CLIP	75.16	61.63	62.75	62.18
BLIP	70.42	65.32	69.25	67.23
OpenVik + BLIP	80.25	72.55	70.61	71.57
Table 6: Grounded situation recognition results (%) of OpenVik enrichment compared with zero-shot baselines.
Qualitative examples. Figure 7 presents a qualitative example of OpenVik-based context enrichment in the grounded situation recognition task. We observed that verbs like "shopping" and "talking" were appropriately enriched with their frequently occurring contexts from the open visual knowledge, leading to a reduced embedding distance between the description and its matching image.
Quantitative results. We assembled a test set of 900 samples from the testing set of GSR that included verbs such as "talking", "filming", and "picking", among others, from a list of 256 words
that can be accurately mapped to extracted visual knowledge, as well as 138 verbs that have a fuzzy match through ConceptNet embedding comparison. The full lists of the exact and fuzzy-matched verbs are detailed in Appendix H. The evaluated metrics include Accuracy, Precision, Recall, and  $F_{1}$ . The results are presented in Table 6. It can be observed that knowledge enrichment significantly outperforms the zero-shot and BLIP baselines. This suggests that the verb-related contexts introduced by OpenVik-generated knowledge are intuitive and greatly assist in understanding the semantics of event verbs, bolstered by related visual information.
5.3 Visual Commonsense Reasoning
Task setting. The goal of visual commonsense reasoning is to predict an answer from four given option candidates for a given image and question. For the baseline approach, we compare the backbone model R2C from the VCR paper [62] and BLIP [27]. In the visual knowledge-enhanced OpenVik Enriched approach, we perform two-level context augmentation, incorporating both entities and relations: (1) we parse the question and options to obtain all  $(S, 0)$  pairs and, for each entity pair, apply the same relation augmentation as in the image retrieval task; (2) for the  $V$  in each option, we enrich the visual context using the same method as illustrated in grounded situation recognition.
Figure 8: An example of OpenVik context enrichment on the VCR task (See Appendix G.3 for more).
Method	Accuracy	Precision	Recall	\( {\mathbf{F}}_{1} \)
R2C	56.66	56.73	56.72	56.72
OpenVik + R2C	59.96	60.01	60.03	60.02
BLIP	62.50	62.50	62.45	62.47
OpenVik + BLIP	67.40	67.54	67.43	67.48
Table 7: Visual commonsense reasoning results  $(\%)$  of OpenVik context enrichment compared with zero-shot baselines.
Qualitative examples. Figure 8 presents an example before and after applying the two-level visual knowledge-based enrichment for visual commonsense reasoning. The results indicate that visual knowledge enhances the correspondence between the correct answer and the image itself.
Quantitative results. We assembled a test set of 939 samples from the validation set of the VCR dataset [62]. Each sample in this test set contains questions and answers with a minimum of five nouns and two relations, guaranteeing an adequate level of information complexity for meaningful engagement with open visual knowledge. The results can be found in Table 7. We observe that the enriched visual knowledge helps especially when solving reasoning questions on humans and their interactions with visually impressive entities, such as "game" in Figure 8. This enhancement results in a performance improvement above  $3.0\%$  over the zero-shot baseline.
This work is the first exploration of a new paradigm of open visual knowledge extraction, which combines an open relational region detector to flexibly pinpoint relational regions and a format-free visual knowledge generator that generates visual knowledge by prompting a multimodality model conditioned on the region of interest. To further enhance the diversity of the generated knowledge, we explore two distinct data enhancement techniques. Extensive knowledge evaluations underscore the correctness and uniqueness of our extracted open visual knowledge, and the consistent improvements observed across various visual reasoning tasks highlight the real-world applicability of OpenVik.
While our approach has been shown effective in various scenarios, its performance at larger scales or on more diverse datasets remains to be studied. Future work could investigate its effectiveness across a broader range of tasks and contexts. Also, the current model requires fine-tuning for the visual knowledge extractor. Developing a model that can generalize well with prompt tuning or demonstration augmentation could be another interesting direction for future work.
Carl Yang was supported by the National Institute Of Diabetes And Digestive And Kidney Diseases of the National Institutes of Health under Award Number K25DK135913.
[57] Yue Yu, Kexin Huang, Chao Zhang, Lucas M Glass, Jimeng Sun, and Cao Xiao. Sumgnn: multi-  
typed drug interaction prediction via efficient knowledge graph summarization. Bioinformatics, 2021.
[58] Yue Yu, Rongzhi Zhang, Ran Xu, Jieyu Zhang, Jiaming Shen, and Chao Zhang. Cold-start data selection for few-shot language model fine-tuning: A prompt-based uncertainty propagation approach. arXiv preprint arXiv:2209.06995, 2022.
[59] Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. arXiv preprint arXiv:2306.15895, 2023.
[60] Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Bridging knowledge graphs to generate scene graphs. In ECCV, 2020.
[61] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. NeurIPS, 2022.
[62] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR, 2019.
[63] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, 2018.
[64] Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung. Aser: A large-scale eventuality knowledge graph. In WWW, 2020.
[65] Yiwu Zhong, Liwei Wang, Jianshu Chen, Dong Yu, and Yin Li. Comprehensive image captioning via scene graph decomposition. In ECCV, 2020.
[66] Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Prompt consistency for zero-shot task generalization. In Findings of EMNLP, 2022.
[67] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
A Details of Data Augmentation with External Knowledge Resources
$\checkmark$  Enhance Relation Recognition: We enriched the relationships between objects parsed from the original knowledge descriptions by leveraging the external resource of ConceptNet. ConceptNet comprises commonly observed entities and their connections, where edge weights signify the reliability and frequency of these relationships. The typical value of edge weights in ConceptNet is 1. To prevent the redundancy of common information and to maintain the validity of the enriched relations, we categorized the relationships based on their weights. Relationships with weights less than 1 were deemed "weak" and those with a weight of 1 were labeled "average". We refrained from using these categories for relation enhancement. Instead, only relationships with weights greater than 1, indicative of high reliability, were employed for augmenting the relations.
$\checkmark$  Boost Entity Perception: On the entity side, we augment complement entities and descriptive information with two external knowledge resources. On one hand, for descriptions with a high TF-IDF+ score, we enrich related entities of the object from ConceptNet to create additional knowledge descriptions. The relatedness is based on the between-word relatedness score provided by ConceptNet and we take the threshold as 0.85. On the other hand, we employ the Commonsense Transformers (COMET) [4] model to enrich related new objects and descriptive information. The COMET model is a language model designed to generate commonsense knowledge and understand causal relationships between descriptions. It is pretrained using the atomic dataset, which consists of structured, crowdsourced knowledge about everyday events and their associated causes and effects. The COMET model can provide neighbor descriptions of the given input of nine different categories of relation. We take the xAttr and oEffect relation categories and augmented the COMET model by formulating the existing knowledge description texts as the input and choose the corresponding category branch during generation for enriching objects and descriptions respectively.
B Dataset Information
Table 8: Dataset statistics.
split	#image	#descriptor	#relation	#subject & object
Train	75,456	832,351	30,241	302,735
Validation	4,871	64,137	5,164	34,177
Test	4,873	62,579	5,031	32,384
The statistic information of our augmented dataset is summarized in Table 8, where split specifies the dataset split, #image indicates the number of images in the split, #descriptor indicates the total number of relational descriptors of the images, #relation is the total number of unique relations in the relational descriptors after dedduplication, and #subject & object is the total number of subjects and objects contained in the description text.
Hyperparameter	Assignment
batch size	4
learning rate optimizer	Adam
Adam epsilon	1e-8
Adam initial learning rate	1e-5
learning rate scheduler	cosine scheduler
Adam decay weight	0.05
Table 9: Hyperparameters for training open relational region detector.
Hyperparameter	Assignment
batch size	4
learning rate optimizer	Adam
Adam epsilon	1e-8
Adam initial learning rate	1e-5
learning rate scheduler	cosine scheduler
Adam decay weight	0.05
α	0.7
φ	0.01
Table 10: Hyperparameters for training format-free visual knowledge generator.