SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hejie Cui $^{1*}$  Xinyu Fang $^{2*}$  Zihan Zhang $^{2}$  Ran Xu $^{1}$  Xuan Kan $^{1}$  Xin Liu $^{3}$
Yue Yu4 Manling Li5 Yangqiu Song3 Carl Yang1
<sup>1</sup>Emory University <sup>2</sup>Tongji University <sup>3</sup>The Hong Kong University of Science and Technology
4 Georgia Institute of Technology 5 Northwestern University
Images contain rich relational knowledge that can help machines understand the world. Existing methods on visual knowledge extraction often rely on the predefined format (e.g., sub-verb-object tuples) or vocabulary (e.g., relation types), restricting the expressiveness of the extracted knowledge. In this work, we take a first exploration to a new paradigm of open visual knowledge extraction. To achieve this, we present OpenVik which consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest. We also explore two data enhancement techniques for diversifying the generated format-free visual knowledge. Extensive knowledge quality evaluations highlight the correctness and uniqueness of the extracted open visual knowledge by OpenVik. Moreover, integrating our extracted knowledge across various visual reasoning applications shows consistent improvements, indicating the real-world applicability of OpenVik.
Knowledge extraction has been widely studied on texts [8, 1, 13, 9] for enhancing logical reasoning [45, 14, 6] and explainable AI [18, 57, 5, 55], and recent studies have explored open knowledge extraction through categorizing seed relations [64, 40] and eliciting from language models [47]. Visual knowledge extraction, on the other hand, captures intricate details like tools, sizes, and positional relationships, which are often difficult to express exhaustively in texts [39, 28, 48, 7]. Yet existing approaches of visual knowledge extraction are either restricted by a fixed knowledge format [52, 63, 20, 22] or the predefined sets of objects/relations [52, 63, 21]. While efficient at capturing interactions between objects, the produced visual knowledge is often limited in richness and confined to a single format, falling short in representing the diverse real-world information that can be complemented by visual data.
In this endeavor, we propose to further explore a new paradigm of open visual knowledge extraction (OpenVik). Specifically, we propose to generate relation-oriented, but format-free knowledge that includes a wider variety of elements, such as descriptions, insertions, and attributes, among others. Drawing inspiration from the wealth of knowledge encapsulated in large models [49, 61, 46], we propose to leverage pre-trained large multimodality models by eliciting open visual knowledge through relation-oriented visual prompting. This approach allows for a more nuanced understanding of visual data, mirroring how humans naturally emphasize certain aspects of visual scenes when perceiving and describing visual information, leading to more flexible visual knowledge extraction.
† Correspondence to: j.carlyang@emory.edu
Our proposed OpenVik framework consists of two modules, an open relational region detector and a format-free visual knowledge generator. It is a unique challenge to detect the regions potentially containing relational knowledge, since traditional region detectors primarily focus on learning predefined object classes. To learn the regression of relational regions, we propose to use free-form knowledge descriptions as supervision and leverage knowledge generation as a training objective. With the detected regions, the remaining question is how to interpret these regions into free-form knowledge. We propose a visual knowledge generator by harnessing the power of language variety enhancement in large pre-trained multimodality models. Specifically, we prompt them to generate knowledge descriptions of any formats and condition the generation on the detected relational regions.
However, establishing a new paradigm of open visual knowledge extraction is challenging due to the absence of comprehensive and diverse training data. Existing datasets sources such as scene graphs [51, 24], dense captions [20], and dense relational subsets [22] often exhibit a long-tail distribution biased to more prevalent relations and entities [44]. Brute-force merging of these datasets could exacerbate the distribution bias inherent in the data. To alleviate the bias, we propose two diversity-driven data enhancement strategies based on an adapted TF-IDF+ score, involving random dropping and data augmentation with external knowledge resources. These strategies optimize data distributions and richness, thus fostering diverse open visual knowledge extraction.
We implement extensive evaluations to assess the quality and utility of the open visual knowledge extracted by OpenVik, encompassing: 1) directly evaluating the performance of knowledge generation; 2) engaging human evaluators for a multi-faceted assessment of in-depth knowledge quality; and 3) comparing the open visual knowledge extracted with OpenVik with existing knowledge sources, such as non-parametric knowledge from the ConceptNet knowledge graph, and parametric knowledge from the GPT-3.5 large language model. Furthermore, the utility of the extracted open visual knowledge is validated through its integration with several common applications that require visual understanding, including text-to-image retrieval, grounded situation recognition, and visual commonsense reasoning. These applications demonstrate consistent improvements, affirming the practical utility of OpenVik.
Visual knowledge extraction. Recent advancements in knowledge extraction have extended from being purely text-driven to incorporating images [11, 29]. VisKE [39] is designed to verify relations between pairs of entities, e.g., eat(horse, hay). Scene graphs, which locate objects in the image and identify visual predicates between subjects and objects in a triple format, e.g., (man, on, chair), are extensively studied for vision understanding [52, 63, 60]. A recent work OpenSGG [17] extends SGG to open-vocabulary objects, enabling the relation prediction for unseen objects. Other studies have explored caption-like formats, like dense captioning [20] with a set of object-centric descriptions across regions, and relational captioning [22] focusing on relational information between objects. Despite these advancements, existing methods either adhere to a pre-defined format and vocabulary or are constrained by the biased distribution of training sets. This highlights the pressing need for a format-free approach in visual knowledge extraction with knowledge diversity.
Large model prompting. Recently, large language and multimodality models have exhibited remarkable successes in capturing commonsense knowledge across various tasks, especially facilitating few-shot [15, 53, 25, 58] and zero-shot learning [23, 66, 59]. The potential of prompt-based learning for pre-trained vision-language models [2, 37, 42] has been explored for handling diverse data types across multiple modalities, such as images and texts, with improved performance in tasks including image classification [33, 67], segmentation [32] and visual question answering [16]. Leveraging the substantial information encapsulated within these pre-trained multimodality models to extract explicit knowledge can enrich existing resources, potentially laying the groundwork for advances in interpretability research and mitigating the hallucination issue associated with large models [19, 10].
In this section, we introduce our new paradigm and two key model design novelty featuring OpenVik, relation-oriented multimodality model prompting and diversity-driven data enhancement.
Figure 1: The overview of OpenVik. The left orange and purple panels illustrate key components of relation-oriented multimodality model prompting: open relational region detector and format-free visual knowledge generator. The right green one depicts diversity-driven data enhancement strategy. OpenVik is designed to extract relation-oriented format-free open visual knowledge with novel entities, diverse relations, and nuanced descriptive details.
3.1 Open Visual Knowledge Extraction
Given a dataset  $\mathcal{D} = \{(\mathcal{I}_i, \mathbf{T}_i, \mathbf{U}_i)\}_{i=1}^M$  consisting of  $M$  samples,  $\mathcal{I}_i$  is the  $i$ -th image (such as the input image in Figure 1),  $\mathbf{T}_i = \{\mathcal{T}_j\}_{j=1}^{n_i}$  is a set of  $n_i$  region descriptions (such as "the boat on water" in Figure 1),  $\mathbf{U}_i = \{\mathcal{U}_j\}_{j=1}^{n_i}$  is the set of  $n_i$  relation-oriented visual regions, where each  $\mathcal{T}_j$  corresponds to a visual region  $\mathcal{U}_j \in \mathbf{U}_i$  in image  $\mathcal{I}_i$ . The goal of our open visual knowledge discovery is to train a model  $\mathcal{M}$  capable of producing a set of format-free knowledge descriptions (such as "large boat docked at pier" in Figure 1) given any image  $\mathcal{I}_k$  during the inference stage.
3.2 Relation-Oriented Multimodality Model Prompting
The overall architecture of OpenVik is shown in Figure 1. It comprises two modules: an open relational region detector  $\mathcal{M}_v$  and a format-free visual knowledge generator  $\mathcal{M}_t$ . The two modules are learned separately during training with our diversity-enhanced data (Section 3.3) and combined to produce format-free visual knowledge at inference. Specifically, the relational region detector  $\mathcal{M}_v$  takes an image  $\mathcal{I}_i$  as the input and learns to select a flexible number of relational regions  $\mathbf{U}_i = \left\{\left(\mathcal{U}_j\right)\right\}_{j=1}^{n_i}$  that captures object interactions, each corresponding to a description  $\mathcal{T}_j$  in  $\mathbf{T}_i$ ; the visual knowledge generator  $\mathcal{M}_t$  generates format-free knowledge descriptions by prompting and fine-tuning the multimodality model with the guidance of detected visual region  $\mathcal{U}_j$ . All notations for the region detector and knowledge generator are detailed in Table 9 and Table 10, respectively.
Table 1: Notations for open region detector.
Notation	Meaning
Ii	input image of the relational region detector
Uj	relation-centric box label
Ui	set of relation-centric boxes of an image
Tj	region description of a box
Ti	set of region descriptions of the an image
LRD	region regression loss supervised by union regional boxes
LK	knowledge generation loss supervised by GT relational knowledge
Lv	the overall objective of the relational region detector
Table 2: Notations for knowledge generator.
Notation	Meaning
Ii	the input image of the knowledge generator
Ta,Tb	two regional knowledge descriptions of one same image
Ni	the number of generated knowledge descriptions of an image
φ	hyper-parameter controlling the penalty slightly different sequences
LMLE	the language modeling loss of the generation decoder
LV	inter-sequence information variety regularizer
α	weight hyper-parameter balancing generation accuracy and variety
l	the overall objective of the knowledge generator
Open relational region detector. Although existing object detection algorithms have been widely recognized for their efficiency in object detection, they are usually restricted to object-centric visual regions in a predefined set, and thus cannot directly capture open relational information with a single box. Detecting regions containing relational knowledge remains to be a challenge. We make two adaptations on the object detection FasterRCNN [38] to train the open relational region detector:
- Region Regression: we change the original object-centric region labels to our newly created relation-centric box labels, denoted as  $\mathbf{U}_j$ . The foreground of each relation-centric region label  $\mathcal{U}_j$  is created by taking the union of the object-level bounding boxes of the entities, i.e., boat, water, contained in a ground truth region knowledge description  $\mathcal{T}_j$ . This forms the region regression loss  $\mathcal{L}_{\mathrm{RD}}$ .
- Knowledge Supervision: To assist with the refinement of the bounding box, we replaced the object-centric label classification in traditional object detectors with knowledge supervision. A pre-trained generator is finetuned to create the regional description grounded to the given region. This is supervised by the cross-entropy loss  $\mathcal{L}_{\mathrm{K}}$  with region description  $\mathbf{T}_j$ .
The training objective  $\mathcal{L}_l$  of the relational region detector is formulated as below, where  $\mathcal{L}_{\mathrm{RD}}$  is the regional regression loss and  $\mathcal{L}_{\mathrm{K}}$  is the knowledge supervision loss,
$$
\mathcal {L} _ {v} = \mathcal {L} _ {\mathrm {R D}} + \mathcal {L} _ {\mathrm {K}}. \tag {1}
$$
Format-free visual knowledge generator. OpenVik provides better knowledge grounding by conditioning the generator on the detected relational region, leading to a reasoning-driven generation. Specifically, the detected bounding box (such as the box containing "boat" and "pier" on the far left) is utilized as a visual prompt when fine-tuning the visual knowledge generator. The model architecture of the knowledge generator is built upon a combined large multimodality model, which composes a pre-trained vision transformer ViT-B [12] and the image-grounded text decoder of BLIP [27]. The two modules are jointly trained on a generic image-text paired dataset comprising over 14 million entries and fine-tuned on the image captioning task, which delivered state-of-the-art performance.
In our visual knowledge generator, the decoder takes the ViT visual representation of the entire image as input and leverages the detected regional mask as a binary visual prompt. This prompt aids in filtering out the background and directing attention toward the relational foreground. The generation of format-free knowledge from the decoder is supervised by the language modeling loss  $\mathcal{L}_{\mathrm{MLE}}$ , which further refines visual attention during the knowledge generation process. As a result, our approach facilitates the production of format-free outcomes that extend beyond the conventional sub-verb-object form. Besides, to improve information variety, we introduce an amplifying penalty factor for highly similar knowledge generation. For any two generated sequences  $T_{a}$  and  $T_{b}$  describing image  $\mathcal{I}_i$ ,
$$
\mathcal {L} _ {\mathrm {V}} = \frac {1}{N _ {i}} \sum_ {N _ {i}} \operatorname {R e L U} (- \log (1 - (s (T _ {a}, T _ {b}) - \phi))), \tag {2}
$$
where  $N_{i}$  is the number of generated knowledge of image  $\mathcal{I}_i$ ,  $s(T_a, T_b)$  indicates the semantic cosine similarity, and  $\phi$  is a hyper-parameter set as 0.01 controlling the penalty on sequences with only slight difference (e.g. "dog chasing the man" and "dog licking the man") to be relatively small.
The training objective  $\mathcal{L}_l$  of the format-free visual knowledge generator is formulated as
$$
\mathcal {L} _ {l} = \alpha \times \mathcal {L} _ {\mathrm {M L E}} + (1 - \alpha) \times \mathcal {L} _ {\mathrm {V}}, \tag {3}
$$
where  $\alpha$  is a weighting hyper-parameter we set as 0.7. The trained relational region detector and visual knowledge generator are combined during inference. Given any image  $\mathcal{I}$ , the open relational region detector first detects a flexible number of open relations regions of interest, then each detected region  $\mathcal{R}$  is passed to the format-free visual knowledge generator, where a relation-oriented format-free knowledge phrase (such as "flying jet leaving behind smoke" in Figure 1) is generated to describe the given visual focus subarea  $\mathcal{R}$  of the image. To further encourage within-sequence language variety during inference, we leverage the contrastive decoding strategy from [43], which improves over nucleus sampling and beam search.
3.3 Diversity-driven Data Enhancement
The training data for relational knowledge extraction usually exhibits a long-tail distribution, where more prevalent but simple relations such as in, on, and wear dominate the training set [44]. Consequently, the model trained with such a biased dataset may render limited, and repetitive knowledge. As a remedy, we propose two data enhancement techniques to optimize the data distribution. As the foundational measure for given relation  $r$ 's importance, we design a grid TF-IDF+ score  $S_{r}$  [34, 54]:
$$
\mathcal {S} _ {r} = \left(\log \left(\frac {N}{1 + f _ {r} * \alpha_ {1}}\right)\right) ^ {\alpha_ {2}}, \tag {4}
$$
where  $N$  is the total number of knowledge phrases in the datasets,  $f_{r}$  is the number of occurrences of the relation  $r$ ,  $\alpha_{1}$  and  $\alpha_{2}$  are the grid scales whose values are selected based on  $f_{r}$ .
Random dropping on low-quality data. We first remove repeated knowledge descriptions in the same image and then randomly drop descriptions that contain frequently occurring yet meaningless
relations with a low  $S_{r}$  (e.g., "people on ground") from the original dataset. Specifically, if the  $S_{r}$  of the relation in a description is relatively low, i.e., 0.4, we remove it at a random dropping rate of 0.5. This process repeats for all descriptions in an image until the remaining set is 0.6 times the size of the original training set. Consequently, the training data bias is mitigated by removing low-quality data.
Data augmentation with external knowledge resources. For the relations with high TF-IDF+ scores, we leverage external knowledge resources from both non-parametric (i.e., ConceptNet [41]) and parametric (i.e., COMET [4]) knowledge resources to promote diverse knowledge generation [56].  $\checkmark$  Enhance Relation Recognition: For each training description with a high TF-IDF+ score, we perform semantic parsing to get all the objects and complement additional relations (e.g., "rest" in Figure 1) between each pair of them by mapping the nodes and retrieving edges from the ConceptNet. Each retrieved knowledge triplet is converted to a knowledge phrase and added to the training set for generator training. With this introduced external knowledge, the knowledge generator ultimately yields a more robust and detailed representation of the underlying visual information of objects. This, in turn, bolsters the relation recognition of the visual knowledge generator.  $\checkmark$  Boost Entity Perception: For the description with the highest-scored TF-IDF+ relation given each image, we also leverage ConceptNet to enrich similar objects (e.g., "jet") to the original object (e.g., "plane"). Additionally, we further introduce new entities (e.g., "smoke" in Figure 1) and attribute descriptions (e.g., "blue") by prompting the pre-trained attribute commonsense branch of the COMET model (Refer to Appendix A for more details). The entity-based enrichment potentially helps in boosting entity understanding and at the same time enhances the occurrence of important but rare relations in the training set.
Our training data are built based on Visual Genome [24] and its relation-enhanced version Dense Relational Captioning [22]. Each sample includes an image identified by a unique ID and a set of relational descriptors describing interactions among objects in the image. Specifically, each relational descriptor includes the full description text, the subject and object names contained in the description text, the relation between them, as well as the bounding box coordinates of the subject and object. The dataset statistic information is summarized in Table 8 in the Appendix B.
Our model is implemented in PyTorch [35] and trained on two Quadro RTX 8000 GPUs. The open relational region detector is initialized from the ResNet50-FPN backbone, then finetuned for another 20 epochs with the relational bounding box. The model detects a maximum of 30 bounding boxes for each image with the highest confidence to avoid misleading noises. The format-free visual knowledge generator is initialized from  $\mathrm{BLIP}_{\mathrm{base}}$  with the basic ViT-B/16 and finetuned for 20 epochs. Full details on learning parameters can be referred to in Appendix C.
In this section, we directly evaluate the extracted open visual knowledge from OpenVik from two perspectives: (1) knowledge generation performance with traditional generative metrics and in-depth knowledge quality assessment; (2) comparison with existing knowledge sources. Besides, ablation studies are conducted to study the influence of diversity design on the generated knowledge and data.
4.1 Evaluation on Generated Knowledge
Generation performance. To directly evaluate the visual knowledge generator, we compare the knowledge generated by OpenVik with a variety of baselines, including scene graph generation [52, 63, 44, 17] (of which Ov-SGG employs an open vocabulary), dense relational captioning [22], and region captioning [20, 65, 27, 26]. Evaluation metrics are traditional language generation measures such as BLEU, ROUGE-L, and METEOR. The results, displayed in the left side of Table 3, reveal that OpenVik outperforms captioning-based approaches and yields results on par with the best scene graph generation baseline. These findings underscore the effectiveness of the format-free visual knowledge generator through relation-oriented prompting of the large multimodality model.
In-depth knowledge quality. To more thoroughly evaluate the quality and richness of the format-free visual knowledge extraction, beyond simply evaluating it as a language generation model with the
Table 3: Knowledge comparison of OpenVik and baselines on performance and in-depth quality  $(\%)$
Method	Generation Performance	In-Depth Knowledge Quality
BLEU↑	ROUGE-L↑	METEOR↑	Validity↑	Conformity↑	Freshness↑	Diversity↑
Closed/Open Scene Graph Generation
IMP [52]	0.075	0.123	0.118	0.800	0.823	0.676	0.316
Neural Motifs [63]	0.229	0.283	0.273	0.822	0.767	0.667	0.349
UnbiasSGG [44]	0.217	0.258	0.194	0.739	0.733	0.666	0.357
Ov-SGG [17]	0.167	0.210	0.183	0.712	0.633	0.693	0.413
Dense Relational Captioning
MTTSNet+REM [22]	0.240	0.226	0.228	0.897	0.852	0.754	0.375
Region Captioning
DenseCap [20]	0.248	0.245	0.196	0.883	0.843	0.790	0.543
Sub-GC [65]	0.272	0.263	0.221	0.892	0.871	0.795	0.547
BLIP [27]	0.264	0.266	0.252	0.886	0.855	0.760	0.531
BLIP2 [26]	0.275	0.285	0.257	0.892	0.871	0.766	0.535
Open Visual Knowledge Extraction
OpenVik	0.280	0.283	0.250	0.907	0.883	0.809	0.619
limitation of training data, we incorporate four additional metrics [31], which delve into an in-depth quality evaluation of the extracted visual knowledge from four distinct perspectives:
- Validity  $(\uparrow)$ : whether the generated visual knowledge is valid to human.
- Conformity ( $\uparrow$ ): whether the generated knowledge faithfully depicts the scenarios in the images.
- Freshness  $(\uparrow)$ : the novelty of the knowledge, i.e., the proportion not present in the training set.
- Diversity  $(\uparrow)$ : the language variance between a randomly sampled pair of knowledge pieces.
Among the four metrics, both the validity and conformity metrics involve human annotators. We randomly selected 100 images as the evaluative subset. Details regarding the scoring guidance and the interface provided to the annotators can be found in Appendix D. The remaining metrics, i.e., freshness and diversity, are calculated automatically. The in-depth knowledge quality evaluation results are displayed in the right part of Table 3, where the average pairwise Cohen's  $\kappa$  on human evaluation results is 0.76 (good agreement). The findings demonstrate that trained with the diversity-enhanced datasets, the format-free visual knowledge extracted by OpenVik significantly outperforms other types of baselines in terms of all four metrics. The improvement of diversity, in particular, reaches  $14\%$  relatively compared with the inference results from the second runner DenseCap, indicating the advantage of OpenVik in generating rich and comprehensive visual knowledge.
4.2 Comparison with Existing Knowledge Sources