Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting
Authors: Hejie Cui
Xinyu Fang
Zihan Zhang
Ran Xu
Xuan Kan
Xin Liu
Yue Yu
Manling Li
Yangqiu Song
Carl Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Visual Knowledge Extraction
Multimodality
Large Model Prompting
TL;DR: The proposed OpenVik takes a first exploration to a new paradigm of open visual knowledge extraction, that can generate format-free visual knowledge by prompting a multimodality model with relational visual prompts.

Abstract: Images contain rich relational knowledge that can help machines understand the world. Existing methods on visual knowledge extraction often rely on the pre-defined format (e.g., sub-verb-obj tuples) or vocabulary (e.g., relation types), restricting the expressiveness of the extracted knowledge. In this work, we take a first exploration to a new paradigm of open visual knowledge extraction. To achieve this, we present OpenVik which consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest. We also explore two data enhancement techniques for diversifying the generated format-free visual knowledge. Extensive knowledge quality evaluations highlight the correctness and uniqueness of the extracted open visual knowledge by OpenVik. Moreover, integrating our extracted knowledge across various visual reasoning applications shows consistent improvements, indicating the real-world applicability of OpenVik.

Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting
Hejie Cui
Xinyu Fang?*
Zihan Zhang²
Ran Xu'
Xuan Kan'
Xin Liu
Yue Yuu
Manling Li
Yangqiu Song
Carl Yang
Emoryy University
Tongji University
The Hong Kong University of Science and Technology
Georgia Institute Technology Northwestern University
Abstract
Images contain rich relational knowledge that can help machines understand the world. Existing methods visual knowledge extraction often rely the pre- defined format (e.g.. sub-verb-obj tuples) or vocabulary (e.g., relation types), restricting the expressiveness of the extracted knowledge. In this work, take a first exploration to a new paradigm of open visual knowledge extraction, To achieve this, we present OpenVik which consists of an open relational region detector to detect regions potentially containing relational knowledge and visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest. We also explore two data enhancement techniques for diversifying the generated format-free visual knowledge. Extensive knowledge quality evaluations highlight the correctness and uniqueness of the extracted open visual knowledge by OpenVik. Moreover, inte- grating our extracted knowledge across various visual reasoning applications shows consistent improvements, indicating the real-world applicability OpenVik.
Introduction
Knowledge extraction has been widely studied on texts g
for enhancing logical reason-
ing [5 14 6 and explainable AI [8, 37 (5). and recent studies have explored open knowledge
extraction through categorizing seed relations (64 40) and eliciting from language models [47].
Visual knowledge extraction, on the other hand, captures intricate details like tools, sizes, and positional relationships, which are often difficult to express exhaustively
texts 39
28
48 7].
Yet existing approaches of visual knowledge extraction are either restricted by a fixed knowledge While efficient at capturing interactions between objects, the produced visual knowledge is often limited in richness and confined to a single format, falling short in representing the diverse real-world information that can be complemented by visual data.
format [52] 63 20 22 or the predefined sets of objects/relations [52] 63
In this endeavor, we propose to further explore new paradigm open visual knowledge extraction (OpenVik). Specifically, we propose to generate relation- oriented,, but format-fred knowledge that includes a wider variety of elements, such as descriptions, insertions, and attributes, among others. Drawing inspiration from the wealth of knowledge encapsulated in large models [49
61 46 we
propose to leverage pre-trained large multimodality models by eliciting open visual knowledge through relation-oriented visual prompting. This approach allows for a more nuanced understanding of visual data, mirroring how humans naturally emphasize certain aspects of visual scenes when perceiving and describing visual information, leading to more flexible visual knowledge extraction.
'These authors contributed equally to this work. "oorespponeence to: j.carlyang @ emory.edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Our proposed OpenVik framework consists of two modules, an open relational region detector and format-free visual knowledge generator. It is a unique challenge detect the regions potentially containing relational knowledge, since traditional region detectors primarily focus on learning predefined object classes. To learn the regression of relational regions, we propose to use free-form knowledge descriptions as supervision and leverage knowledge generation as training objective. With the detected regions, the remaining question is how to interpret these regions into free-form knowledge. We propose visual knowledge generator by harnessing the power of language variety enhancement in large pre-trained multimodality models. Specifically,, we prompt them to generate knowledge descriptions of any formats and condition the generation on the detected relational regions.
However, establishing a new paradigm of open visual knowledge extraction is challenging due to the absence of comprehensive and diverse training data. Existing datasets sources such as scene
graphs [51. 24], dense captions [20], and dense relational subsets [2 often exhibit a long-tail
distribution biased to more prevalent relations and entities [44]. Brute-force merging of these datasets
could exacerbate the distribution bias inherent in the data. To alleviate the bias, we propose two diversity-driven data enhancement strategies based on an adapted TF-IDF+ score, involving random dropping and data augmentation with external knowledge resources. These strategies optimize data distributions and richness, thus fostering diverse open visual knowledge extraction.
We implement extensive evaluations assess the quality and utility of the open visual knowledge ex- tracted by OpenVik, encompassing: 1) directly evaluating the performance of knowledge generation; 2) engaging human evaluators for mulli-faacteed assessment of in-depth knowledge quality; and 3) comparing the open visual knowledge extracted with OpenVik with existing knowledge sources, such as non--arammee con-paraettric knowledge from the ConceptNet knowledge graph, and parametric knowledge from the GPT-3.5 large language model. Furthermore, the utility of the extracted open visual knowledge is validated through its integration with several common applications that require visual understanding, including text-to-image retrieval, grounded situation recognition, and visual commonsense reasoning. These applications demonstrate consistent improvements, affirming the practical utility of OpenVik.
2
Related Work
Visual knowledge extraction. Recent advancements in knowledge extraction have extended from being purely text-driven to incorporating images [1] 29). VisKE [39] is designed to verify relations between pairs of entities, e.g., eat(horse, hay). Scene graphs, which locate objects in the image and identify visual predicates between subjects and objects in a triple format, e.g., (man, on, chair), are extensively studied for vision understanding 52
63 60). A recent work OpenSGG [17] extends SGG to open-vocabulat objects, enabling the relation prediction for unseen objects. Other studies have explored caption-lit Mike formats, like dense captioning [20] with a set of object-centrie descriptions across regions, and relational captioning 22] focusing relational information between objects. Despite these advancements, existing methods either adhere to pre-defined format and vocabulary or are constrained by the biased distribution of training sets. This highlights the pressing need for format-free approach in visual knowledge extraction with knowledge diversity.
Large model prompting. Recently, large language and multimodality models have exhibited re- markable successes in capturing commonsense knowledge across various tasks, especially facilitating 66, 59). The potential of prompt-based learning 42 has been explored for handling diverse data types
few-shot [5 53 25 58 and zero-shot learning 23
for pre-trained Dision-langusl across multiple modalities, such as images and texts, with improved performance in tasks including image classification B3 [67], segmentation 32 and visual question answering [16]. Leveraging the substantial information encapsulated within these pre-trained multimodality models extract explicit knowledge can enrich existing resources, potentially laying the groundwork for advances in interpretability research and mitigating the hallucination issue associated with large models [19 10]
models 2 37
3 Method
In this section, we introduce our new paradigm and two key model design novelty featuring OpenVik, relation-oriented multimodality model prompting and diversity data enhancement.
2

---

selected original data
relation enkanced
entily enlanced
+Relation
Lx
Training Data
bout rests waier
the water
+Eatity
Detector
plane air
the in blue
ERD
crond stand on ground
mokke path in
Open Relational Region Detector
Inference
TF-IDF+
Augmeutation with Extemal
I
Resources 3
Low
Random Dropping
KG
1
Format-free Visual Knowledge Generator
Diversity-Drive Data Enhancement
Prompting
Large Multimodaiity
Open Visual Knowledge
Model
larget boat docked at
OpenVik
LNLE
Generated Knowledge Similarity
Rhing lnveing holiind smokle
Relation-Oriened Regions
busiding blocki across from brove blue water standing
high aboue water
Figure I: The overview of OpenVik. The left orange and purple panels illustrate key components of relation-oriented multimodality model prompting: open relational region detector and format free visual knowledge generator. The right green one depicts diversity- data enhancement strategy. OpenVik designed to extract relation-oriented format- free open visual knowledge with novel entities_ diverse relations and nuanced descriptive details
3.1 Open Visual Knowledge Extraction
Given a dataset D = {[,, T, U M consisting of M samples, I, is the i-th image (such as the input image in Figure 1h. T, = ( is set of n, region descriptions (such as "the boat on water" in Figure []. U (4, m is the set of relation-oriented visual regions, where each Tj corresponds to visual region Ug E U, in image I. The goal of our open visual knowledge discovery is to train a model M capable of producing a set format-free knowledge descriptions (such "large boat docked at pier" in Figure I given any image Ia during the inference stage.
3.2 Relation-Oriented Multimodality Model Prompting
The overall architecture of OpenVik is shown in Figure [ It comprises two modules: open relational region detector M, and a format-free visual knowledge generator M, The two modules are learned separately during training with our diversity-enhancee data (Section 3.3) and combined to produce format-free visual knowledge inference. Specifically, the relational region detector M, takes image I, as the input and learns to select flexible number of relational regions U. = ) that captures object interactions, each corresponding to a description T, in T,; the visual knowledge generator M, generates format-free knowledge descriptions by prompting and fine-tuning the multimodality model with the guidance of detected visual region Uj All notations for the region detector and knowledge generator are detailed Table 9 and Table TO respectively.
Table 1: Notations for open region detector.
Table 2: Notations for knowledge generator.
Notation
Meaning
Notation
Meaning
isput image the elatioall detector relains-camii label relution cencie
inpur of knowledge
T N.
two rigional knowletj desemptions one
image
the number genemned know ledge descripions contolling penalty slighlyy diffewen sepuenees the lnngugg modttling loss the ginzationn dosoder information varity rpulaiie
descipption of descriptions
Cans Ev
regresion supersiseal GT celationa kaowlodge knowkedge generation the relational region detecnoe the overall
regional
weight weight hypes-pmmntter belancing genendioon accurasy and variety lee overall objestive the knowlodge generatue
4
Open relational region detector. Although existing object detection algorithms have been widely recognized for their efficiency in object detection, they are usually restricted to object-centric visual regions in predefined set, and thus cannot directly capture open relational information with single box. Detecting regions containing relational knowledge remains to be a challenge. We make two adaptions the object detection FasterRCNN
38 to train the open relational region detector:
Region Regression: we change the original object-centric region labels to our newly created relation- centric box labels, denoted as Uy. The foreground of each relation-centre centric region label Ui is created by taking the union of the levect-level bounding boxes of the entities, i.e., boat, water, contained in a ground truth region knowledge description Ty. This forms the region regression loss CRD-
3

---

Knowledge Supervision: To assist with the refinement of the bounding box, we replaced the object-centric label classification in traditional object detectors with knowledge supervision. A pre-trained generator is finetuned to create the regional description grounded to the given region. This is supervised by the cross-entropy loss Lx with region description T--
The training objective Ci of the relational region detector is formulated as below, where LRD is the regional regression loss and Lx is the knowledge supervision loss,
L
LRD + Lx.
Format-free visual knowledge generator. Open Vik provides better knowledge grounding by conditioning the generator on the detected relational region, leading to a reasoning-driver generation. Specifically, the detected bounding box (such the box containing "boat" and "pier" on the far left) utilized as a visual prompt when fine-tuning the visual knowledge generator. The model architecture of the knowledge generator is built upon a combined large multimodality model, which composes pre-trained vision transformer ViT-B [12) and the image-grounded text decoder of BLIP [27]. The two modules are jointly trained on a generic image-text paired dataset comprising over 14 million entries and fine-tuned on the image captioning task, which delivered state-of-the-ar performance.
In our visual knowledge generator, the decoder takes the ViT visual representation of the entire image as input and leverages the detected regional mask as a binary visual prompt. This prompt aids in filtering out the background and directing attention toward the relational foreground. The generation of format-free knowledge from the decoder is supervised by the language modeling loss LMLE. which further refines visual attention during the knowledge generation process. As result, approach facilitates the production of format-free outcomes that extend beyond the conventional sub-verb-ob) form. Besides, improve information variety, we introduce an amplifying penalty factor for highly similar knowledge generation. For any two generated sequences Ta and T. describing image I,
Lv
Σ N
ReLU (- log (1 (s (Ta, Tb) - -))),
N
where N, is the number of generated knowledge of image I, (Ta, indicates the semantic cosine similarity, and & is a hyper-paramen parameter set as 0.01 controlling the penalty on sequences with only slight difference (e.g. "dog chasing the man" and "dog licking the man") to be relatively small.
The training objective L. of the format-free visual knowledge generator formulated as
L = a x LMLE + (1 (1-a) a) x Lv.
(3)
where a a weighting arper-parameter we set as 0.7. The trained relational region detector and visual knowledge generator are combined during inference. Given any image I, the open relational region detector first detects flexible number of open relations regions of interest, then each detected region R is passed to the format-fred visual knowledge generator, where a relation-oriented format-free knowledge phrase (such as "flyng jet leaving behind smoke" in Figure [ is generated to describe the given visual focus subarea R. of the image. To further encourage within-sequence language variety during inference, we leverage the contrastive decoding strategy from 43 which improves over nucleus sampling and beam search.
3.3 Diversity-driven Data Enhancement
The training data for relational knowledge extraction usually exhibits a long-tail distribution, where more prevalent but simple relations such as in, on, and wear dominate the training set [44]. Conse- quently, the model trained with such a biased dataset may render limited, and repetitive knowledge. As remedy, we propose two data enhancement techniques to optimize the data distribution. As the foundational measure for given relation r's importance, we design grid TF-IDF+ score S,
N
S, = (log(
1 01 ))0²
(4)
where N is the total number knowledge phrases in the datasets, f. is the number of occurrences of the relation r, on and a2 are the grid scales whose values are selected based on fc-
Random dropping on low-quality data. We first remove repeated knowledge descriptions in the same image and then randomly drop descriptions that contain frequently occurring yet meaningless

---

relations with low S, (e.g., "people on ground") from the original dataset. Specifically, if the S, of the relation in a description relatively low, ie.. 0.4, we remove it at random dropping rate of 0.5. This process repeats for all descriptions in an image until the remaining set is 0.6 times the size of the original training set. Consequently, the training data bias is mitigated by removing low-quality data.
Data augmentation with external knowledge resourees. For the relations with high TF-IDF4 scores, we leverage external knowledge resources from both non-parametric (i.e., ConceptNet 41) and
parametric (i.e., COMET 4) knowledge resources to promote diverse knowledge generation [56]. / Enhance Relation Recognition: For each training description with high TF-IDF+ score, we perform semantic parsing to get all the objects and complement additional relations (e.g.. "rest" in Figure between each pair of them by mapping the nodes and retrieving edges from the ConceptNet. Each retrieved knowledge triplet is converted to a knowledge phrase and added to the training set for generator training. With this introduced external knowledge, the knowledge generator ultimately yields more robust and detailed representation of the underlying visual information of objects. This, turn, bolsters the relation recognition of the visual knowledge generator. Boost Entity Perception: For the description with the highest-scored TF-IDF+ relation given each image, also leverage ConceptNet to enrich similar objects (e.g., "jet") to the original object (e.g., "plane"), Additionally, we further introduce new entities (e.g.. "smoke" in Figure I and attribute descriptions (e.g.. "blue") by prompting the pre-trained attribute commonsense branch of the COMET model (Refer to Appendix Afror more details). The entity- based enrichment potentially helps in boosting entity understanding and at the same time enhances the occurrence of important but rare relations in the training set.
3.4 Implementation Details
Our training data are built based on Visual Genome (24 and its relation-enhanced version Dense Relational Captioning [22]. Each sample includes an image identified by unique ID and set of relational descriptors describing interactions among objects in the image. Specifically, each relational descriptor includes the full description text, the subject and object names contained in the description text, the relation between them, as well as the bounding box coordinates of the subject and object. The dataset statistic information is summarized in Table X
in the Appendix B
Our model is implemented in PyTorch (35 and trained on two Quadro RTX 8000 GPUs. The open relational region detector is initialized from the ResNet50-FPN backbone, then finetuned for another 20 epochs with the relational bounding box. The model detects a maximum of 30 bounding boxes for each image with the highest confidence to avoid misleading noises. The format-free visual knowledge generator is initialized from hhsee with the basic ViT-B/16 and finetuned for 20 epochs. Full details on learning parameters can be referred to in Appendix
4 Evaluation
In this section, we directly evaluate the extracted open visual knowledge from OpenVik from two perspectives: (1) knowledge generation performance with traditional generative metrics and in-depth knowledge quality assessment; (2) comparison with existing knowledge sources. Besides, ablation studies are conducted to study the influence of diversity design the generated knowledge and data.
4.1 Evaluation on Generated Knowledge
Generation performance. To directly evaluate the visual knowledge generator, we compare the
knowledge generated by OpenVik with a variety of baselines, including scene graph generation [52]
63 44
(of which Ov-SGG employs an open vocabulary), dense relational captioning (2), and
region captioning 20 65 27 26). Evaluation metrics are traditional language generation measures such as BLEU, ROUGE-L, and METEOR. The results, displayed in the left side of Table 3 reveal that OpeNViK outperforms captioning based approaches and yields results on par with the best scene graph generation baseline. These findings underscore the effectiveness of the format-free -reee visual knowledge generator through relation-oriented prompting of the large multimodality model.
In-depth knowledge quality. To more thoroughly evaluate the quality and richness of the format- free visual knowledge extraction, beyond simply evaluating it language generation model with the
5

---

Table 3: Knowledge comparison of OpenVik and baselines on performance and in-depth quality (%)
Generation Performance
In-Depth Knowledge Quality
Method
BLEUT
ROUGE-L1
METEORT
Validity
Conformity?
Preshness?
Diversity
Closed/Open Scene Graph Generation
IMP [22
0.075
0.123
0.118
0,800
0.823
0.676
0.316
Neural Motifs 66
0.229
0.283
0.273
0.822
0.767
0.667
0.349
Unb
0.217
0.258
0.194
0.739
0.733
0.666
0.357
Ov-SGG
0.167
0.210
0.183
0.712
0.633
0.693
0.413
Dense Relational Captioning
RT 22
0.240
0.226
0.228
0.897
0.852
0.754
0.375
Region Captioning
DenseCap 20
0.248
0.245
0.196
0.883
0.843
0.790
0.543
Sub-GCI 65
0.272
0.263
0.221
0.892
0.871
0.795
0.547
BLIP 22
0.264
0,266
0.252
0.886
0.760
BLIP2 26
0.275
0.285
0.257
0.892
0.871
0.766
0.535
Open Visual Knowledge Extraction
OpenVik
0.280
0.283
0.250
0.907
0.883
0.809
0.619
limitation of training data, we incorporate four additional metrics 31 which delve into an in-depth quality evaluation of the extracted visual knowledge from four distinct perspectives:
Validity (f): whether the generated visual knowledge is valid human.
Conformity (f): whether the generated knowledge faithfully depicts the scenarios in the images. Freshness (f): the novelty of the knowledge, i.e., the proportion present in the training set.
Diversity (f): the language variance between randomly sampled pair of knowledge pieces.
Among the four metries, both the validity and conformity metrics involve human annotators. We randomly selected 100 images as the evaluative subset. Details regarding the scoring guidance and be found in Appendix D The remaining metrics, i.e.,
the interface provided the annotators freshness and diversity, are calculated automatically. The in-depth knowledge quality evaluation results are displayed in the right part of Table 3] where the average pairwise Cohen's 's human evaluation results 0.76 (good agreement). The findings demonstrate that trained with the diversity- enhanced datasets, the format-free visual knowledge extracted by OpenVik significantly outperforms other types of baselines in terms of all four metrics. The improvement of diversity, in particular, reaches 14% relatively compared with the inference results from the second runner DenseCap, indicating the advantage of OpenVik in generating rich and comprehensive visual knowledge.
4.2 Comparison with Existing Knowledge Sources
Non-parametric Knowledge
Parametric Knowledge
Know Source
Examples
Non Non-paramerric Knowledge
dog lsA miimal (C (ConosptNet): concepenct IsA knowledge graph (C (ComoppNee); dog Histhoorrt back (C (ConcepiNett: dog and brown für covering black (OpenVik)
Relerioss
Mbstort
compuner keyboadd (ConeppNeet:; key
computer (OpenViic)
Parametric Knowledge
people uinn light yellow sign in comer (both;; back seat attached bike (both:
illuniuue mem, (LLM)
Amtruuss Bekngggge
Descrptrve Likrecto
Open Visual Knowledge (OpenVik)
three loyer cake on table: blue trash can full of garbage next to beown dresser: bize bot siring beside a sneaky ganger (OpenVikk)
Speial Ditais, Metton Dynorits, Scasc Inight
people wearing fachionable block hats are skiing: baby elephants walking around advestunnus wood, (QpenVik)
the light shining Goo brigla black backgrond, hanging fan above sill shelf: (OpenVik)
Open
bnown chair the bockggrond
Figure 2: The Venn diagram of knowledge comparison between the open visual knowledge from OpenVik with the non-parametric knowledge from existing knowledge graph (i.e., ConceptNet) and parametric knowledge from large language model (i.e., COMET).
We compare the extracted visual knowledge with the non-par parametric knowledge the existing knowledge graph (KG) and the parametric knowledge from the large language model (LLM). The comparison insights from the three knowledge resources are shown in the Venn Diagram in Figure 2
Compare with non-parametric knowledge. We take ConceptNet 41 as the representative in the comparison with parammrrettric knowledge. To map the knowledge generated by OpenVik to ConceptNet, parse the knowledge into triplets and associate the endpoints of these triplets with

---

nodes in ConceptNet. Then we calculate the similarity of embeddings between the parsed relation and all the edge relations among the mapped nodes in ConceptNet. If the similarity score exceeds a predetermined threshold, i.e., 0.75, we consider the mapping successful. As illustrated in Figure 2 we observe that compared with the non-parametric knowledge in KG, the extracted visual knowledge captures richer and more meaningful spatial details, e.g.. "three layer cake on table", and motion dynamics, e.g., "baby elephants walking around adventurous wood".
Compare with parametric knowledge. We compare with parametric knowledge contained in LLM by prompting the gpt-3.5-turbe model with the object information in the image. The prompt template used is detailed in Appendix E The mapping process follows the approach mentioned earlier. It is found that compared with the parametric knowledge in LLM, the extracted visual knowledge exhibits unique fine- thee-gr visual details, e.g., "red sticker on fence". and provides precise scene information, e.g., "the light shining from bright black background".
4.3 Ablation Study
The influence on knowledge quality with information variety regulariza- tion and data strategies. We con- ducted ablation studies to evaluate the effectiveness of the information vari- cty regularizer, Lv, and our diversity- driven data enhancement strategies. This involves an in-depth assessment of knowledge quality on the same eval- uation subset. The results are presented in Figure 3 It is evident from the results that our proposed information variety design primarily impacts freshness and diversity, without compromising validity and conformity. For the freshness, the omission of data augmentation for entities and relations results in the most significant performance degradation. This implies the crucial role these strategies play in infusing novel knowledge into the generation process. As for diversity, the most notable changes in metries are observed when the Ly and random dropping are removed. The strategy for augmenting entities and relations also plays valuable role in enriching diversity.
C.
Rag
Fau
DX
6x
0.5
0.5
Validity
65
Conformity
0.5
Preshness
0.50
Divensiyy
Figure 3: The influence of information variety regularization
and diversity-driven data enhancement strategies.
Ablation of the pre-training for the open relational region detector. We conducted a comparison of the outcomes when loading pre-trained detector backbone versus training the detector from scratch, as shown by the yellow bar in Figure 3 Results demonstrate noticeable decrease in both knowledge diversity and freshness, which indicates the importance of loading the pre-trained model for region detection. This may be because omitting the pre-training step of the FaNterRCNN model tends to result in the detection of more overlapping regions, which in turn causes the drop.
The influence on dataset diversity with data strategies. We conduct direct analysis of the knowledge diversity of the existing datasets and our diversity-enhanced one, compared with the visual knowledge generated from OpenVik. The findings, presented in Table 4 show that the dliversity-rriver data enhancement strategies significantly boost knowledge diversity. Trained with this enhanced data, OpenVik can extract visual knowledge that exhibits greater diversity than that found in the Visual Genome and Relational Caps, indicating the advantage of OpenVik to format-free visual knowledge generation and its ability to yield richer knowledge diversity.
Table 4: Diversity of existing and enhanced datasets and generated knowledge from OpenVik.
Training Dataset Relutional Caps ( Diversity Enhanced (Ours)
Generate Knowledge
Metries
Visuel Genome
OpenVik (Ows)
Diversity
0.589
0.604
0.632
0.619
4.4
Case Study
We present two case studies in Figure 4 (See Appendix F for more) to showcase the format-free visual knowledge generated by OpenVik, in comparison Visual Genome (Scene Graph and Region Description) and Relational Caps. Contrary to the rigidity of scene graphs, which strictly adhere to a
Fmmeedding are produced by ConceptNet API: http://tthhccccoooommmmeeemcmmme http:://ppaatommoopppai.co wtt.://www....ooooooo/ee//ga.aa.
7

---

predefined format, OpenVik can generate knowledge with flexible semantic structure, not strictly bound the sub-verb-obj format (e.g.. "blue post attached wall with white letter"). Examples of this adaptability are highlighted in red. When compared to dense region descriptions, the relational knowledge extracted by OpenVik offers a deeper understanding of the multiple entity interactions within an image. In comparison to Relational Caps, which mainly focus on interactions between two objects, OpenVik significantly broadens the diversity of relation with vivid verbs (e.g., "attached to", "adorning"), Moreover, it introduces novel entities (e.g., "post", "mane") and enriches the knowledge representation with nuanced details (e.g., "full of", "striped") that are missed by Relational Caps.
CR/
Vinual Genom Scene Graph:
Visual Genome-Scene Graple -hair, eaebm, cat graze>
CAY
<dtrikk In, oooler> sornngg, in,
KOJ
<lecuer od, luilding
eays on, zebra>
FAN
<ilem, 65, latll>
grond>
C A
Visual Genome-l Seame--eeg Region Descriptions: oranges green red wriling on white sign drink rod cooler
Visual G Region Descriptions black white stsppd leg light shiming on the zehra this line of black luirr two zehras graaing the grass
A
lhing
oranges
VienVikk
Open lk:
post attiched wall
intter
strppd mane belongs gracing Pebm
open
suowy ground
Relational Caps
zobm smiped cating grom grass
Relational Caps:
L wood box Billia Willewnt lase of onange
covrred ortagges wond thing
stope adoming leg
up gasiing eye sating zebea geazing sebce green grass the muzale of grazing zoben
2 whate Senner oz bralding with leterr a
on snow covered oranges
brown monse growing bohind
D Blue box sitting beside ttesky garage green leaves on snow-orvered oranges
grass everpwchon sarmonnd standing zehra
D intsered will
leeves
rnd writing on white sign
Baack ucse above grees livels gress
Figure 4: Case study the extracted open visual knowledge from OpenVik. Examples of format-free knowledge are highlighted in red. Compared with VG and Relational Caps, OpenVik performs better at capturing novel entities, broadening object interactions with diverse relations and enriching the knowledge representation with nuanced descriptive details
Note that we observe the unbalanced and noisy distributions within the training data can lead to errors in the knowledge produced. Viewing hallucinations erroneous inferences based on input, the inaccuracies observed in OpenVik and similar baselines often stem from detection erros.. These errors are typically caused by data biases that incorrectly associate features with a specific class or label. We further two illustrative failure cases in Figure 5 For example, a "black speaker by flat tv" is generated, although the speaker is not present in the imagc-possibly reflecting common co-occurrences within the dataset. Similarly, a ladder in the right figure has been misidentified as towel, leading to the erroneous description of a "blue towel hanging from dry shower". The key to mitigating such incorrect inference is identifying the cofounder feature of class labeling.
Visual Genome-Sonne Graph: oninduw, in, screepo cays, of cat>
Ganome Graph: spainl in. frame spectore, on, wall> <winčow, bathrcom>> <zurtiin, over, windoee
<leyboard, on, Japtap> <xs, beside, laptogo
Visual Genome-Tegion Descriptions laptoç
Visual Genome-Regiin Region Descriptions elephani baity elephantinn grass
beside the laptop picturss on wall behand cat the cat ls brown
Grass withis the enclosur big rock on the ground
OpenVik:
OpenVilk:
while ost sitling our compute
Relational Caps:
frimed picture langing on wall recessed lightng celling Mue lowel lissgng from dry stowet the sunlight coming through window
Retational Capes blue panting metal frame shamgoo bottleon blse jar window
Mlccs speniare by Eat Mark wire playpet into compuler the picture zn screen
whate beside black laptop window behttnd est
Figure 5: Examples of incorrectly knowledge resulting from distribution bias are highlighted
Application
This section explores whether the extracted open visual knowledge from OpenVik can bolster reasoning and inference capabilities in multimodality downstream tasks by augmenting a baseline in the challenging zero-shot setting.
5.1 Text-to-Image Retrieval
Task Setting. In the text-to-image retrieval task, a given caption is matched a large set of candidate images, with the most relevant image returned the result. Adopting the challenging zero-shot
8

---

setting, we generate the visual representation v and textual representation t of the given image I and caption T using a pre-trained clip-retrieval model [
The baseline involves the image and text and the fine-tuned model from BLIP (26).
embedding similarly based on zero-shot CLIP Retrieval
K
To explore the potential of the extracted visual knowledge from OpenVik, we enrich the given caption T with related contexts derived from the extracted visual knowledge. Specifically, for each query caption, we parse the caption to extract all subject-object pairs (s, o) with the NLTK parser. Then s and o are mapped to the open visual knowledge, where knowledge phrases that contain relations occurring more than 30% of the time between $ and o are enriched the original caption T.
room and dining room area with furniture A living room with a big table next
Original
Rocatle1
Recalles
Rerailsh
Ang
to book shelf. A living
decorated with
ZS-CIIP OpenVik 25-CLIP
40.55
36 16
65.07
75.29
84.53
78.06
66.12
00.10
modem theme A living room with wooden floors
HUP
63
86.90
91.10
00.17
and furniture. The large
has wooden table
B RIIP
68
87.71
91:00
BFAI
with chairs and couch.
Eariched text: big table in room. decorated living room with wooden furniture. brown couch room. on wonden wooden booksieelf with books next table.
Table 5: Text-to-image retrieval results (%)
shelf. shelf couch.
of OpenVik enrichment compared with zero-shot baselines.
Figure 6: An example of OpenVik enrichment on text- to-image retrieval (See Appendix G.T for more).
Qualitative examples. Figure pressnts an example of OpenVik- --baeed visual knowledge enrichment on captions. By incorporating related contexts from the generated open visual knowledge, the enriched captions convey more precise visual details, which enhances the alignment for text-image alignment.
Quantitative results. We curated a subset of 680 images from the testing set of the MS-COCO dataset containing parsed knowledge with at least eight nouns. This ensures an adequate degree of enrichment is achieved through the use of OpenVik. Standard image retrieval metrics, i.e., Recall @ 1/5/10/ and Avg, are employed to evaluate the performance. The results are presented in Table 5 It is evident that relational context enrichment leads to the average correction of more than 6.0% of the initial zero-shot, highlighting the practical benefits of extracted visual knowledge in visual reasoning tasks.
5.2 Grounded Situation Recognition
Task setting. The event type prediction for the grounded situation recognition task to predict the best match from predefined 504 event types (36 based on the image. We convert each candidate event verb into a description T: "An image of <verb>" " for image description matching. Similarly to text-to-image retrieval, we include zero-shot CLIP and the fine-tuned model from BLIP as baselines.
To enrich with contextual knowledge from OpenVik, for each given verb v, we find its nearest synonym in the extracted open visual knowledge and enrich the text description with the most common knowledge phrase containing it, regularized by the objects present in the image. Instead of directly concatenating the retrieved knowledge triplets to the original textual description, we employ an additive decomposition strategy: the similarity s(I, v) of the candidate verb with respect to the given image I is calculated as s(I, v) D( EdeD(e) ((I, v), where D(v) is the set of descriptors, including the original description and the enriched ones, and represents the single log probability that descriptor d pertains to the image I.
Verb: talking
Method
Accuracy
Precision
Recall
Original text: This is an image of talking. Enriched man talking small white telephone, adult male with white shirt talking on chatty cellphone, the man waving arms talking on phone.
25-CLIP
53.14
42.54
45 19
43.62
Cpen + ZS-CLIP
75:16
61.63
62.75
62.18
BLIP
70.42
65.32
69.25
67.23
Vpw - RUP
8025
72.85
70.1
7137
Table 6: Grounded situation recognition results (%) of OpenVik enrichment com-
Figure 7: An example of OpenVik context enrichment on task GSR (See Appendix G.2 for more).
pared with zero-shot baselines.
Qualitative examples. Figure 7 presents a qualitative example of OpenVik-based context enrichment in the grounded situation recognition task. We observed that verbs like "shopping" and "talking" were appropriately enriched with their frequently occurring contexts from the open visual knowledge, leading to a reduced embedding distance between the description and its matching image.
Quantitative results. We assembled a test set of 900 samples from the testing set of GSR that included verbs such "talking", "filming", and "picking", among others, from a list of 256 words

---

that can be accurately mapped to extracted visual knowledge, as well as 138 verbs that have a fuzzy match through ConceptNet embedding comparison. The full lists of the exact and fuzzy-matcher verbs are detailed in Appendix H The evaluated metrics include Accuracy, Precision, Recall, and Fi. The results are presented in 6 It can observed that knowledge enrichment significantly outperforms the zero-shot and BLIP baselines. This suggests that the verb-related contexts introduced by
knowledge are intuitive and greatly assist in understanding the semantics of event verbs, bolstered by related visual information.
5.3 Visual Commonsense Reasoning
Task setting. The goal of visual commonsense reasoning is predict an answer from four given option candidates for a given image and question. For the baseline approach, we compare the backbone model R2C from the VCR paper [62 and BLIP 27). In the visual knowledge-enhancee OpenVik Enriched approach, we perform two-level context augmentation, incorporating both entities and relations: (1) we parse the question and options to obtain all (S, 0) pairs and, for each entity pair, apply the same relation augmentation as in the image retrieval task; (2) for the V in each option, we enrich the visual context using the same method as illustrated in grounded situation recognition.
Question: Is Passon
game?
Method
Accuracy
Precision
Recall
engaged game:
Yes,
person
R2C
56,66
56.73
engiged game ponon walking neway
DpenVi2 RIC
60:01
losing. pesson ongaged game. diustrated person gume.
B1
62.50
Openvi2
67.54
No. not scally crjoying penan engaged is game perser erjoy coletration Yes, looks he pood land persrn engaged is game the person is warching left hand.
Table 7: Visual commonsense reasoning results (%) of OpenVik context enrich- ment compared with zero-shot baselines.
Answer: You he looks like be has good hand.
Figure 8: An example of OpenVik context enrichment on the VCR task (See Appendix G.3 for more).
Qualitative examples. Figure 8 presents an example before and after applying the two-level visual knowledge-basee enrichment for visual commonsense reasoning. The results indicate that visual knowledge enhances the correspondence between the correct answer and the image itself.
Quantitative results. We assembled a test set of 939 samples from the validation set of the VCR dataset [62]. Each sample in this test set contains questions and answers with minimum of five nouns and two relations, guaranteeing an adequate level of information complexity for meaningful engagement with open visual knowledge. The results can be found in Table We observe that the enriched visual knowledge helps especially when solving reasoning questions on humans and their interactions with visually impressive entities, such as "game" in Figure g This enhancement results in performance improvement above 3,0% over the zero-shot baseline.
Conclusion, Limitations, and Future Work
This work is the first exploration of a new paradigm of open visual knowledge extraction, which combines an open relational region detector to flexibly pinpoint relational regions and format-fres visual knowledge generator that generates visual knowledge by prompting multimodality model conditioned on the region of interest. To further enhance the diversity of the generated knowledge, we explore two distinct data enhancement techniques. Extensive knowledge evaluations underscore the correctness and uniqueness of our extracted open visual knowledge, and the consistent improvements observed across various visual reasoning tasks highlight the real-world applicability of OpenVik.
While our approach has been shown effective in various scenarios, its performance at larger scales or on more diverse datasets remains to be studied. Future work could investigate its effectiveness across broader range of tasks and contexts. Also, the current model requires fine-tuning for the visual knowledge extractor. Developing model that can generalize well with prompt tuning or augmentatin augmentation could be another interesting direction for future work.
demonstration
7
Acknowledgments
Carl Yang was supported by the National Institute Of Diabetes And Digestive And Kidney Diseases of the National Institutes of Health under Award Number K25DK
10

---

References
[1] Harith Alani, Sanghee Kim, David E Millard, Mark J Weal, Wendy Hall, Paul H Lewis, and Nigel R Shadbolt. Automatic ontology-basee -aaggy knowledge extraction from web documents. IEEE Intell Syst, 18:14-21.
[2] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
[3] Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build clip retrieval 2022.
system with them.
[4] Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. COMET: commonsense transformers for automatic knowledge graph construction. In ACL, 2019.
[5] Aaron Chan, Jiashu Xu, Boyuan Long, Soumya Sanyal, Tanishq Gupta, and Xiang Ren. Salkg: Learning from knowledge graph explanations for commonsense reasoning. In NeurIPS, 2021.
[6] Xuelu Chen, Ziniu Hu, and Yizhou Sun. Fuzzy logic based logical query answering on knowledge graphs. In AAAI, 2022.
[7] Hejie Cui, Rongmei Lin, Nasser Zalmout, Chenwei Zhang, Jingho Shang, Carl Yang, and Xian Li. PV2TEA: Patching visual modality to textual-established information extraction. In ACL, 2023.
[8] Hejie Cui, Jiaying Lu, Yao Ge, and Carl Yang. How can graph neural networks help document retrieval: A case study on cord19 with concept map generation. European Conference on IR Research (ECIR), 2022.
[9] Hejie Cui, Jiaying Lu, Shiyu Wang, Ran Xu, Wenjing Ma, Shaojun Yu, Yue Yu, Xuan Kan, Tianfan Fu, Chen Ling, et al. A survey on knowledge graphs for healtheare: Resources, application progress, and promise. ICML Workshop on Interpretable Machine Learning in Healthcare, 2023.
[10] Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may not be faithful: Probing object hallucination in vision-language pre-training. In EACL, 2023.
[11] Yang Ding, Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, and Qi Wu. Mukea: Multimodal knowledge extraction and accumulation for knowledge-base based visual question answering. In CVPR, 2022.
[12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
[13] James Fan, Aditya Kalyanpur, David Gondek, and David Ferrucci. Automatic knowledge extraction from documents. IBM J Res Dev, 56:5-1, 2012.
[14] Mikhail Galkin, Zhaocheng Zhu, Hongyu Ren, and Jian Tang. Inductive logical query answering in knowledge graphs. In NeurIPS, 2022.
[15] Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACI-IJCNLP, 2021.
[16] Jiaxian Guo, Junnan Li. Dongxu Li. Anthony Meng Huat Tiong, Boyang Li. Dacheng Tao, and Steven CH Hoi. From images to textual prompts: Zero-shot vqa with frozen large language models. In CVPR, 2023.
[17] Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Towards open-vocabulary scene graph generation with pased finetuning. In ECCV, 2022.

---

[18] Andreas Holzinger, Peter Kieseberg, Edgar Weippl, and A Min Tjoa. Current advances, trends and challenges of machine learning and knowledge extraction: from machine learning to explainable ai. In CD-MAKE, 2018.
[19] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput Surv, 55:1-38, 2023.
[20] Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. CVPR, 2016.
[21] Xuan Kan, Hejie Cui, and Carl Yang. Zero-shot scene graph relation prediction through commonsense knowledge integration. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD)), 2021.
[22] Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, and In So Kweon. Dense relational captioning: Triple-stream networks for relationship- captioning. In CVPR, 2019.
[23] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurlPS, 2022.
[24] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123:32-73, 2017.
[25] Hunter Lang, Monica N Agrawal, Yoon Kim, and David Sontag. Co-training improves prompt- based learning for large language models. In ICML, 2022.
[26] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
[27] Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping language- image pre-training for unified vision- language understanding and generation. In ICML, 2022.
[28] Manling Li, Alireza Zareian, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare Voss, et al. Gaia: A fine-grained multimedia knowledge extraction system. In ACL, 2020.
[29] Ye Liu, Hui Li, Alberto Darraa Mathias Niepert, Daniel Onoro-Rubio, and David Rosenblum. Mmkg: multi-modal knowledge graphs. In ESWC, 2019.
[30] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[31] Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang. Lianhui Qin. Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. NeurlPS, 2022.
[32] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR, 2022.
[33] Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR, 2023.
[34] Jiaul Paik. A novel tf-idf weighting scheme for effective ranking. In SIGIR, 2013.
[35] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, pigf-peeformance deep learning library. NeurlPS, 2019.
[36] Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. Grounded situation recognition. In ECCV, 2020.
12

---

[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI,, 2021.
[38] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun, Faster r-cnn: Towards real-tima object detection with region proposal networks. NeurIPS, 2015.
[39] Fereshteh Sadeghi, Santosh K Kumar Divvala, and Ali Farhadi. Viske: Visual knowledge extraction and question answering by visual verification of relation phrases. In CVPR, 2015.
[40] Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. Atomic: An atlas of machine commonsense for if-then reasoning. In AAAI, 2019.
[41] Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. AAAI, 2017.
[42] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-ligguistic representations. In ICLR, 2020.
[43] Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In NeurlPS, 2022.
[44] Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In CVPR, 2020.
[45] Komal Teru, Etienne Denis, and Will Hamilton. Inductive relation prediction by subgraph reasoning. In JCML, 2020.
[46] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NeurIPS, 2021.
[47] Chenguang Wang, Xiao Liu, and Dawn Song. Language models are open knowledge graphs. arXiv preprint arXiv:2010.1 1967,, 2020.
[48] Meng Wang. Sen Wang. Han Yang, Zheng Zhang. Xi Chen, and Guilin Qi Is visual context really helpful for knowledge graph? a representation learning perspective. In ACM MM, 2021.
[49] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoe V. Le, and Denny Zhou. Chain-of- of-thought prompting clicits reasoning large language models. In NeurlPS, 2022.
[50] Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training, 2019.
[51] Danfei Xu, Yuke Zhu, Christopher Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR, 2017.
[52] Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing, In CVPR, 2017.
[53] Ran Xu, Yue Yu, Hejie Cui, Xuan Kan, Yanqiao Zhu, Joyce Ho, Chao Zhang, and Carl Yang. Neighborhoodd-regulriizee self-training for learning with few labels. In AAAI, 2023.
[54] Ran Xu, Yue Yu, Joyce Ho, and Carl Yang. suaklyss scientific document classification via retrieval-augmenter multi-stage training. In SIGIR, pages 2501-2503. 2023.
(55) Ran Xu, Yue Yu, Chao Zhang, Mohammed K Ali, Joyce C Ho, and Carl Yang. Counterfactual and factual reasoning over hypergraphs for interpretable clinical predictions ehr. In Machine Learning for Health, 2022.
[56] Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmenta- tion for commonsense reasoning. In Findings of EMNLP, 2020.
13

---

[57] Yue Yu, Kexin Huang, Chao Zhang, Lucas M Glass, Jimeng Sun, and Cao Xiao. Sumgnn: multi- typed drug interaction prediction via efficient knowledge graph summarization. Bioinformatics. 2021.
[58] Yue Yu, Rongzhi Zhang, Ran Xu, Jieyu Zhang, Jiaming Shen, and Chao Zhang. Cold-start data selection for few-shot language model fine-tuning: A prompt-based uncertainty propagation approach. arXiv preprint arXiv:2209 06995, 2022.
[59] Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. arXiv preprini arXiv:2306. 15895, 2023.
[60] Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Bridging knowledge graphs generate scene graphs. In ECCV, 2020.
[61] Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. NeurIPS, 2022.
[62] Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR, 2019.
[63] Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR, 2018.
[64] Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung. Aser: A large-scale eventuality knowledge graph. WWW, 2020.
[65] Yiwu Zhong, Liwei Wang, Jianshu Chen, Dong Yu, and Yin Li. Comprehensive image captioning via scene graph decomposition. In ECCV, 2020.
[66] Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Prompt consistency for zero-shot task generalization. In Findings of EMNLP, 2022.
[67] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
14

---

A Details of Data Augmentation with External Knowledge Resources
Enhance Relation Recognition: We enriched the relationships between objects parsed from the original knowledge descriptions by leveraging the external resource ConceptNer. ConceptNet comprises commonly observed entities and their connections, where edge weights signify the re- liability and frequency of these relationships. The typical value of edge weights in ConceptNet is To prevent the redundancy of common information and maintain the validity of the enriched relations, we categorized the relationships based on their weights. Relationships with weights less than were deemed "weak" and those with a weight of were labeled "average". We refrained from using these categories for relation enhancement. Instead, only relationships with weights greater than indicative of high reliability, were employed for augmenting the relations.
Boost Entity Perception: On the entity side, we augment complement entities and descriptive information with two external knowledge resources. On hand, for descriptions with a high TF- IDF+ score, we enrich related entities of the object from ConceptNet to create additional knowledge descriptions. The relatedness is based on the between-word relatedness score provided by ConceptNet and we take the threshold as 0.85. On the other hand, we employ the Commonsense Transformers (COMET) 4 model enrich related new objects and descriptive information. The COMET model is a language model designed to generate commonsense knowledge and understand causal relationships between descriptions. is pretrained using the atomic dataset, which consists of structured, crowd- sourced knowledge about everyday events and their associated causes and effects. The COMET model can provide neighbor descriptions of the given input of nine different categories of relation. We take the xAttr and oEffect relation categories and augmented the COMET model by formulating the existing knowledge description texts as the input and choose the corresponding category branch during generation for enriching objects and descriptions respectively.
B Dataset Information
Table 8: Dataset statisties.
split
#image
#descriptor
#relation
#subject & object
Train
75,456
832,351
30,241
302,735 34,177
Validation 4.871
64,137
5,164
Test
4.873
62,579
5,031
32.384
The statistic information of our augmented dataset is summarized in Table 8 where split specifies the dataset split, #image indicates the number of images in the split, #descriptor indicates the total number of relational descriptors of the images, #relation is the total number of unique relations in the relational descriptors after deduplication, and #subject & object the total number of subjects and objects contained in the description text.
C Implementation Details
Hyperparameter
Assignment
Hyperparameter
Assignment
batch size
batch size
4 Adam 1e-8
learning rate optimizer
Adam
learning rate optimizer
Adam epsilon
1e-8
Adam epsilon
Adam initial learning rate
le-5
Adam initial learning rate
le-5
learning rate scheduler
cosine scheduler
learning rate scheduler
cosine scheduler 0.05
Adam decay weight
0.05
Adam decay weight
0.7
0.01
Table 9: Hyperparameters for training open relational region detector.
Table 10: Hyperparameters for training format-free visual knowledge generator.
Open relational region detector. The visual feature extraction backbone is constructed upon pre-trained ResNet50-FPN, The detector head incorporates BLIP basc equipped with the essential
15

---

ViT-B/16 for text supervision, using multiple fully connected layers to derive region features. For each candidate region, we engage a regressor to conduct boundary regression on these features. The detector undergoes fine-tuning for 20 epochs using the relational region bounding box dataset and an Adam optimizer
30) The hyperparameters for training are detailed in Table [
Format-free visual knowledge generator. The format-frre visual knowledge generator is initialized from BLIP base. which incorporates the basic ViT-B/16. We fine-tune the generator model for 20 epochs using the same optimizer as the one employed for the region detector. Detailed hyperparameters for the visual knowledge generator can be found in Table I0
D Human Evaluation Guidance and Interface
We perform the human evaluation on two of the four in-depth knowledge quality assessment metrics. We build an interface by referring [50], where raters are presented with a given image and the corresponding knowledge descriptions and are required to choose one from the multiple choice for two questions on whether the knowledge is valid to humans and whether the knowledge description depicts the image. The detailed scoring criteria for Validity and Conformity are provided below:
Validity (t): whether the generated visual knowledge is valid to humans.
0 (Invalid): The knowledge description does not conform to human cognition, rendering it unreliable or misleading to humans.
(Valid): The knowledge description is valid and accurately conforms to human cognition, providing reliable and meaningful knowledge to humans.
Conformity (f): whether the generated knowledge faithfully depicts the scenarios in the images. (Inconsistent): The knowledge description does not faithfully depict the scenarios in the images, showing significant deviations or discrepancies, making it difficult for users to relate the textual information to the visual context.
(Partially Conforming): The knowledge description partially conforms to the scenarios in the images, but there might be minor inconsistencies or missing relevant details.
2 (Moderately Conforming): The knowledge description exhibits moderate level of con- formity with the scenarios in the images, capturing the key aspects and providing coherent descriptions.
3 (Highly Conforming): The knowledge description highly conforms to the scenarios in the images, accurately capturing the details and faithfully representing the visual context.
acconaeyy viliily
Knosledos Desciodiaa/2/5tt elvrr chair neat courder
Rare the vailhhy and contermity of eech kncelooge desscnptionn
Auagrn conkor
this Knowledoo Cospriction valid?
Liita yravise' farkon'' saagen cocingge
(Niol Valie)
(Veld))
mideading
vosi leveling deving th crconii
1a this Kinpelodop Desorictio conform?
paree povida matire teoaight sroutegge ti
0 (Inconeisent Contorming
(Fartally Confoming)
O (Moder rodderatall Conforing)
Invem. thes right moondrannis vitsing iilerie
(righly Conformin)
Moyrealay Ou duniiig soovingg cheur
images
Horde Confering The inoelerg qpden
Figure 9: The human evaluation interface for in-depth knowledge quality evaluation.
A ggreement/valliattonn We use Cohen's 's K as the agreement score to measure potential subjectivity involved in ratings of knowledge quality. Cohen's 's k is a statistic that is used to measure inter-rater reliability for qualitative items and is scaled from -1 (perfect systematic disagreement) to 1 (perfect agreement), where values < 0 as indicating no agreement and 0.01-0.20 as none to slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0,81-1.00 as almost perfect agreement. Our calculated average pairwise Cohen's 's x on human evaluation results from three different raters 0.76, which indicates good agreement.
16

---

E Parametric Knowledge Prompting Template
Given image Z and the corresponding extracted visual knowledge from it based on OpenVik, perform knowledge comparison with parametric knowledge contained LLM by prompting the gpt-3.5-turbo model with the object information contained in the I. The prompt format is shown in followings:
Suppose you are looking an image that contains the following subject and object entities:
Subject list: [Insert the subject names here]
Object list: [Insert the object names here]
Please extract 5-10 condensed descriptions that describe the interactions and/or relations among those entities in the image. Try to elucidate the associations and relationships with diverse language formats instead being restricted to sub-verb-obj tuples.
F
More Case Studies of Open Visual Knowledge from OpenVik
10 shows some other cases the extracted open visual knowledge from OpenVik. In comparison to VG and Relational Caps, OpenVik exhibits superior performance at capturing novel entities expanding object interactions through diverse relations and enriching knowledge repre- sentation with nuanced descriptive details For example for the bottom right image, OpenVik can extract novel entities such as tracks shoe" diverse relations such as sticking out of and nuanced descriptive details such as cold thick with man feet on it brave The generated knowledge with more format-free semantic structure is highlighted in red.
Figure
Genoma-SSenee Graph:
Visual Genome Soue Graphu cocting; on, table>
<wurtan has, Inir> spensu,, west, hal>
<agg, ii, cartone
<lea,, plant>
In, hige
sanas, In, siib>
<uaioo, aes lines>
top Genome-Regiee Descriptions
Region Descriptions
oolonal ogg
man in sui. maz under on umbrolia the people are enjoying their
sevea cesnges on bable
onion stalks that lave been ost bunch of lale table
OpenV lk.
OpenVil::
groen simbbery growing siómaalic
while and letwa egg insde cartboard box the izae bunch jempped uana rope greez lettcee ca coneded table
purple
holding people
Relational Cape:
Relational
green frees distance growing aside green bush su sid building
the people people walking the woman wamting biue joons
buildng
while aggs
white thh cloth has flowers docomted puppl onion cloth
green vegetables table pupple ouica between onsage the oggs carton
U people are loying notl to pel wel
keey Inight
the woman looks bock setoutshment
the woman gleess
Genome- Scene Graph:
Visual Genome-Scene Graphe
<kile in, sky>
<tree, fields
spappla in, water>
<hal, en head> groundo
<boad,, with, person>-
<shitt persoaz-
<duxiow, of, skiee>
Visual Genome-R Descriptions westing woman beach
divot Region Descriptions
werting on the pazts lons on the sungliss ski attached right Soot.
the person standing beach the person in woter
Rolational Cape:
Relational downhill
blue board with
jacket
cloudy
red lus worn head the weeds in snow
clouds sky
OpeaVik
OpenVik lk:
vallow till of mdd and yallow libe
the womann munning flytng lato Bec
Dwiile ground covered with cold Uhick snow
by biane
enjovii fill beach liliing winll
Tying kile in the blse
bros slicking out snow-oiverd oi field C skinng man weaning yellow shoe
orange appormmne
the looking kite
the fresh
young skier las nel hor and amiling
Figure 10: Case studies of open visual knowledge from OpenVik.
17

---

G More Qualitative Examples on Applications
G.1 Text-to-lmage Retrieval
Original lext: Three younng men playing Wii on a projecton
Original tests now porked muieeeylla sittnng in frumt of tall buikiing A stene street with bicycles and motor hikes porked on and people standing suikiings
selevisin. Thrse man luighing at
pictures fiom projoctse A group of pentieman ploping video grenes dimly
people chilling with Nintendo Wit. A group of zien playing a game witl resiove controllens.
couch playing
edewallss
Clayazape of pedestiians enjoyligg
an old European city, a row of bikes and muspll is parked alung the street. Motoreycles and mupels line a sede stront during the people.
city.
Eariched text: group. behind people. men playing, video game, group people, men group playing video game. people playing. people watching gumnz playing game.
Enrtiched made of leading street stone,
stone nest people in crassing street, street city, on side, people riding moter city, moter in line. people in line in city. day as city.
behind people.
Original text: elderly weenan sitting on bench resting. her back while sitting on an omate bench, woman is sitting in hench noar a fersee Older womnn drew wiiting park bench, An old woman sitting bench fereee
Original test: hard of catlle feeding the river's edge. Many' cows
worsu
field. body water standing in grasy amzi next to water, heed of catte is near flack of binls swimminng in the water.
of cantle
Enriched text: wortan siting on bench with
behind Sence.
park. bench drows the hends. fence bohisd park.
Fariched text: hard of camll crooing river hand trvolling by water cattle crossing river, cantle field. river azress field in front of area. water near fheld. water water nead to flock. Binds inside of water. Blcck field.
Original test:
leaning
Ortginal text: white refrigerator freezer simig fhigge kiiche plain fridge the cocner. A refrigemtto in the comerr of kitchen just cff the dining coom comm showing very big fridge and dining table.
fence offering elephant. reaching out
an elephaats tunk near gate. A nan is fasding an elaphant over fence. man handing elephent wtick enclovare reaches
Eariched test: reffigentor has freezer
give elephant something.
refigemncc in comenn refrigenttr in hright ianchen. refrigerator room. refiigeraio table siting kitchen, freezer table, window
Eariched text: man behind lense, man next peeparing
holding enclesure man pointing at something fence imck hchind fance waepral smund lonk fouss bchind daphant. lisse made efalidd. fince sumounds enclosure. trnk of elephant, elephant enclosure.
table. diige kitchen, table kitchen. fridge sable tabl: romn.
Figure 11: Qualitative examples of OpenVik context enrichment on text-to-image retrieval.
Figure [ presents more qualitative examples of OpenVik-baser visual knowledge enrichment on captions. The enriched text based on the objects present in the images themselves, supplemented with additional relationships from our generated visual knowledge in OpenVik. It is shown that the introduced relationships often provide new context information that aligns with the visual content of the images. For example, in the image of an old woman sitting a bench in park, the enriched context information includes the positional relationship between the "bench", "fence", and "park", which provides more comprehensive description of the original image.
G.2 Grounded Situation Recognition
Figure 12 presents more qualitative examples of OpenVik-based context enrichment in the grounded situation recognition (GSR) task. Our context enrichment setting for the GSR task is perform enrichment based on verbs like "shopping" and "carrying". We further restrict the enriched context with the objects contained in the image to avoid noisy enrichment. For example, for the image showing people shopping at market, the enriched knowledge contexts could be "the people shopping al market", "standing person shopping for fruit". The idea to enrich the original description T: "An image of <verb>" with relevant actions and relations with the extracted visual knowledge from OpenVik, which can potentially help in drawing-in in the matched candidates.
G.3 Visual Commonsense Reasoning
Figure 13 presents more qualitative examples of OpenVik- based context enrichment the visual commonsense reasoning (VCR) task. The context enrichment on VCR is performed at two-level,
18

---

Verb: shopping
Verb; licking
Origin text: This is an image shopping,
Original text: This is an image of licking.
Eariched black dog licking food.
Enrich text: the people shopping at market. standing person shopping fruit.
curving Original
Verb: spraying
Original text: This is an image of spraying. Enriched text: the water spraying from fountain. the spraying from spout, water spraying in park.
an image carving.
Enriched text: wood carving in center. man carving wood.
Verb: walking Original text: This is an image walking.
Verb: typing Original
Verb: carrying
Original text:
This image of typing. Enriched text: sitting woman typing on smart open laptop.
This is an image carrying. Enriched text: walking person carrying bag. man carrying hay in the field.
Enriched text:
the person
walking through forest. the people walking on sidewalk.
Figure 12: Qualitative examples of OpenVik context enrichment task GSR.
incorporating both entities and relations: (1) we parse the question and options to obtain all (S, 0) pairs and, for each entity pair, apply the same relation augmentation in the image retrieval task; (2) for the V in each option, we enrich the visual context using the same method as illustrated in GSR. It is shown that unrelated answers the image, thus enlarging the distance between incorrect answers and the question, e.g., the enriched contexts "squating person fixing handy bathroom" for example 3 in Figure [3] At the same time, the knowledge description of the correct answer is enhanced by incorporating information that aligns with the image contents, e.g., the enriched knowledge contexts "sitting people red ground" for example in Figure 13
usually enriched with contexts that not relevant to
H Full List of Filtered Verbs for GSR
We provide the full list of verbs out of the predefined 504 candidates of GSR 36 that can be accurate-matched or fuzzy-matched to extracted visual knowledge in Table [1 based which we compose the testing subset for our evaluation on GSR application in Section 5.2.
19

---

Question: Where Fersont sittng?
Question:
Where
going?
llle is in laberatury.
A Person2 is going into the store. the penson walking into sture.
B He is sitting at bar. the person siting behind seesky burrier.
B Penson2 is gotning into carriage. sitting person inside carriage
In a fort in his house the person walking by light houe. sining
C Persoel is geing the bathrooo equeting person fixing handl bathroom.
ground sittig person red ground.
Personl is poing
play ather conversation Persont is over.
Answrr: D He sitting the ground
Answer: A Personil poing ino the store.
Question: Why is Persont in motion??
Question:
Parsonz
Person14 raining desporaelly stcly.
Personi will speak angelly Biningmiez thas walk
Perses7 clumbing over the the person standing insude white bost.
R Person2 will sit down on chsiel pointing person near giint chair
Penson is walking fast the buthroom. squating person fixing handy buchroom.
C Renonn will feed Boxill the person skate boarding in atbletic howl.
to proteet Ransnnl from threat, Pessell moving forward challenge what could there.
Berson? is going
person hoklnng hox full ocanges. Answer: Persont sit down Shaiel
will open
Amswer: Pesson] climbing over
Question: Where Personll and Person2:
Question: What Parsentl doing here?
Bensni Persond are sitting outode of general store. person walking
He prison serving a prison sentenoe, person writing sentenees.
by store.
He is trying to get infonmation. person gaining information.
Personl and Personi are standing top of trin car, jumpung penson on top white the person walking active top couneer Personl and Persond are an office. walking person outside office.
Pemson is waiter, parsen talking with wailer restaurant.
board. walking
He is existing building. walking person lare buikling.
yellow siting atop track. sliced
Answer: Person] waiter.
Penson and Person are the kitchen, the person earing in hungry kitchen.
Ammer: C Pemun] and Persond are office.
Figure 13: Qualitative examples of OpenVik context enrichment on task VCR.
20

---

Table 11: The full list of filtered verbs for GSR.
Matching Type
The Word List of Event Types
putting, butting, bathing, dusting, rearing, turning, skating, placing. carting, staring, biting, mashing. folding, wetting, sprinkling, branch- ing, drying, standing, flaming, taxiing, performing, cireling, molding, parachuting, glowing, fishing, drinking, speaking, pawing, blocking, milking, racing, stripping, potting, spinning, eating, making, kicking, catching, lacing, urinating, sleeping, pressing, buttering, shearing, slid- ing, hiking, glaring, dipping, swimming, shopping, slicing, shelling, wag- ging, grilling, crafting, raining, clawing, splashing, rubbing, snowing, breaking, guarding, clipping, sewing, braiding, telephoning, buttoning, waiting, serving, picking, camping, leaning, working, kissing, wrapping. trimming, tripping, pasting, soaring, driving, knceling, pumping, col- oring, lighting, training, ducking, bowing, arching, cooking, checking, pushing, flipping, rocking, cresting, cleaning, reading. nailing, stitching, building, climbing, covering, shelving, attaching, calming, selling, glu- ing, dyeing, lapping, photographing, peeling, sprouting, licking, display- ing, combing, stacking. planting, fastening, buying, mopping, burning, erasing, measuring, dining, tattooing, gardening, decorating, clearing, fixing, weeding, pulling, feeding, watering, crowning, shaking, dripping, emptying, typing. chasing, poking, leaping, pouring, hanging, sniffing, piloting, falling, overflowing, resting, crashing, carving, ballooning, wad- ing, loading, shaving, boarding, pinning, rowing, juggling, shoveling, hugging, throwing, calling, singing, carrying, walking, writing, crouch- ing, floating, painting, opening, tying, riding, strapping, dialing, saying, bubbling, signing, camouflaging, operating, leading, laughing, parading, skiing, drawing, gnawing, celebrating, spreading, filling, giving, running, smelling, plowing, helping, brushing, scooping, adjusting, wrinkling, steering, biking, smiling, spraying, boating, paying, chewing, stuffing, clinging, landing, wheeling, talking, scoring, teaching, jogging, pitching, flapping, tipping, scrubbing, sitting, surfing, stirring, competing, drum- ming, jumping, filming, dancing. waxing, hitting, recording, baking, waving, washing, signaling, chopping, stretching, rafting, microwaving, phoning. lifting, swinging, releasing, ramming, towing, packing. hauling. frying (244 words)
Accurate
educating, marching, spanking, descending, smearing, heaving, cram- ming, inflating, stooping, inserting, squeezing, tugging, tilting, moisten- ing, swarming, subduing, waddling, winking, flexing, punching, attack- ing, nuzzling, sprinting, sucking, puckering, sketching, rotting, video- taping, complaining, tuning, locking, hurling, pricking, arranging, con- structing, slapping, sweeping, restraining, dousing, frisking, twisting, wringing, hoisting, immersing, shredding, blossoming, igniting, spying, offering, pouting, confronting, docking, assembling, prying, grinning, sharpening, pruning, disciplining, nipping, coaching, nagging, storming, handcuffing, apprehending. bouncing, elenching. taping, distributing, striking, studying, plunging, curling, aiming, sowing, grinding, rinsing, punting, mowing, hitchhiking, skipping. leaking, providing, hunching, spoiling, kneading, burying, foraging, lathering, vaulting, ejecting, mend- ing, pinching, deflecting, ascending, peeing, bothering, repairing, ped- aling, ailing, fueling, skidding, scraping, soaking, grimacing, scolding, spitting, knocking, crushing, bandaging, saluting, fording, stumbling, discussing, raking, launching. whirling, fetching, brawling, retrieving, snuggling, exercising, colliding, stroking, whipping, tilling, betting, farming, browsing, examining, dropping, barbecuing, ignoring, asking, flinging, perspiring, embracing, slipping, flicking, smashing, arresting, lecturing, tearing, gasping, applying, counting, spilling, dragging, recov- ering, practicing, scratching, shooting, packaging, hunting, stinging (154 words)
Fuzzy
21


Reviews:
[{"review_id": "s1FR88KJBq", "reviewer": "Reviewer_PUUs", "summary": "This paper proposes a new paradigm called open visual knowledge extraction and designs a framework OpenVik to generate format-free knowledge instead of pre-defined format knowledge. The authors also present two data enhancement technologies to ensure the diversity of knowledge. Moreover, the paradigm could also be integrated with other applications to jointly boost the performance.", "questions": "1. If the supervision is the generated description, how could it guarantee the correspond label for the region that contains the same objects and relations? Is it a few-shot task?\n2. More detail of generator should be provided. What is the input of generator? Is the single region or the whole image?", "limitations": "Yes. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to understand. \n2. The proposed open relational region detector is interesting.\n3. The evaluation for in-depth knowledge quality is significant.\n", "weaknesses": "1. The originality of the work is incremental. Indeed, both open relational region detector and format-free visual knowledge generator are minor modification existing models. \n2. The experiments presented in this paper are insufficient. The comparison baselines should be more explored because the task of this paper is quite similar to region captioning task, such as [1], [2].\n3. The word ‘open’ is confusing in the title. The paper could not tackle the open-world knowledge. It is still based on the close-set of objects.\n4. The implementation of open relational region detector lacks the detailed information, such as the update of bounding box. The label set raises another question. \n\nReferences\n[1] Zhong, Yiwu, et al. \"Comprehensive image captioning via scene graph decomposition.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16. Springer International Publishing, 2020.\n[2] Ghosh, Shalini, et al. \"Generating natural language explanations for visual question answering using scene graphs and visual attention.\" arXiv preprint arXiv:1902.05715 (2019).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***The originality of the work is incremental and made minor modifications to existing models***\n\n**A**: We would like to clarify several distinct differences between OpenVik and existing models:\n- **Open relational region detector**: Existing detectors often focus on locating objects, while OpenVik is trained to directly detect regions that capture **interactions** and **abstract semantic structures**, such as vivid verbs like “attached to” and nuanced details like “full of”. Some alternative region detectors often need additional visual controls, such as mouse clicks [1, 2] or language controls [3] on a combination of predefined sets of textual properties. OpenVik saves this additional input with its automatic visual grounding ability, which thus improves **knowledge diversity and freshness** significantly. \n\n- **Knowledge generator**: One big difference is that OpenVik is the reasoning-driven generation. Existing regional captioners or knowledge generators often rely on object-level annotation, where the text decoder generates descriptions based on the localized object set [4]. **This leads to the model working like a bag-of-word and a lack of deep semantic understanding.** OpenVik provides **better knowledge grounding** by conditioning the generator on the detected relational region. It includes the ability to **automatically discover the types of interest** that are not only salient but also benefit downstream relational reasoning, as proved in various downstream tasks in Section 5. \n \n- **Language diversity**: Training such a new paradigm for open relational visual knowledge extraction is not trivial. The lack of diverse training data and distribution bias present significant challenges. As shown in Section 4.3 and Figure 3, the **diversity-driven data enhancement** strategies put forth in our work can effectively optimize knowledge richness. They are also generalizable to other models or backbones.\n\nIn light of these, our work goes beyond minor modifications of existing models. We have sought to tackle the challenging problem of open visual knowledge acquisition and deep semantic grounding. The proposed generic techniques can also be applied to other models and datasets.\n\n[1] Pont-Tuset, Jordi, et al. \"Connecting vision and language with localized narratives.\" ECCV, 2020.\\\n[2] Yan, Kun, et al. \"Control image captioning spatially and temporally.\" ACL. 2021.\\\n[3] Deng, Chaorui, et al. \"Length-controllable image captioning.\" ECCV, 2020.\\\n[4] Wu, Jialian, et al. \"Grit: A generative region-to-text transformer for object understanding.\" arXiv preprint arXiv:2212.00280 (2022).\n\n\n>***The comparison baselines should be more explored since the task is quite similar to region captioning***\n\n**A**: We thank the reviewer for the insightful comments. Please refer to our added results under ***Q1*** in the global response.\n\n>***The paper could not tackle open-world knowledge***\n\n**A**: In OpenVik, both objects and relations go beyond a pre-defined set. This is achieved through the design of two core modules:\n\n- Relational detector: The relational detector performs region-level detection instead of object level detection. This is done within a single, holistic detection box supervised by knowledge beneficial for reasoning, allowing for deep semantic and combinational understanding that is not constrained by specific object categories.\n\n- Generative knowledge process: The subsequent knowledge generation is designed to be generative and format-free, not restricted to mere classification or generation around a closed set of objects. This flexibility allows it to capture information in a more open-ended manner, reflecting the true diversity and complexity of real-world visual information.\n\nAdditionally, the diversity strategies proposed help to mitigate distribution bias in training data, encouraging the generation of novel knowledge, including in zero- and few-shot scenarios. \n\n>***The implementation of open relational region detector***\n\n**A**: Thanks for pointing out the areas that need further clarification. The training objective of our relational region detector consists of two components: region regression and knowledge supervision, as detailed in lines 106-110.\n\n- **Region Regression**: This part is guided by our newly generated region labels, denoted as $U_j$. As detailed in lines 108-109, the foreground of these relation-centric region labels is created by taking the union of the object-level bounding boxes of the entities, such as *boat*, *water*, contained in a ground truth region knowledge description $T_j$. This forms the region regression loss $L_{RD}$.\n\n- **Knowledge Supervision**: To assist with the refinement of the bounding box, we have replaced the object-centric label classification found in traditional object detectors with knowledge supervision (line 110). A pre-trained generator is fine-tuned to create the regional description for the given region. This is supervised by the ground truth description $T_j$ with loss term $L_K$.\n\nThe combined training objective for the relational region detector is $L_v = L_{RD} + L_K$. We recognize that this notion (in Figure 1) was not clearly explained. We are grateful to the reviewer for pointing it out, and we will make this clear in the final version of the paper.\n\n>***What is the input of generator? Is the single region or the whole image?***\n\n**A**: Thank you for the inquiry. OpenVik can handle multiple detected relational regions obtained from the detectors for a given image. As detailed in lines 118-120, each of these regions is processed individually and serves as an input to the generator. Along with the specific region, the generator also receives the ViT representation of the entire image. The detected region is utilized as a binary attention mask, which helps to filter out the background and concentrate solely on one relational foreground at a time."}, {"review_id": "heem1iLL1y", "reviewer": "Reviewer_hCst", "summary": "This paper that introduces a new paradigm of open visual knowledge extraction called OpenVik. This method generates format-free knowledge by prompting a large multimodality model with detected regions of interest. The proposed framework consists of an open relational region detector and a format-free visual knowledge generator. The paper highlights the limitations of existing approaches to visual knowledge extraction and demonstrates the correctness and uniqueness of the extracted open visual knowledge by OpenVik. The extracted knowledge is integrated across various visual reasoning applications, showing consistent improvements and indicating the real-world applicability of OpenVik.", "questions": "1. There are many other notions in Figure 1, such as L_k, can the authors explain all of them or list all the equations?\n2. What is binary mask (line 113) in the format-free visual knowledge generator?\n3. What is the training loss of the method? It is not clear to me, since there are many notions in Figure 1 but not appearing in the paper.\n4. Can this method be scale to larger training dataset, or other backbones?\n5. In the evaluation of in-depth knowledge quality, the authors mentioned they randomly selected 100 images. How about the compared method, did they use the same images or same number of images? How many raters or annotators in the studies?\n6. In the application section, are the results of OpenVik zero-shot or fine-tuning? The implement details are not clear in the applications, and need to compare with more other methods.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel Approach: The paper introduces a new paradigm, OpenVik, for open visual knowledge extraction. This system generates format-free knowledge by prompting a large multimodality model with detected regions of interest, which is a novel approach in the field.\n2. Comprehensive Evaluation: The paper provides a thorough evaluation of the generated knowledge, using traditional generative metrics and in-depth knowledge quality assessment. This comprehensive evaluation helps validate the effectiveness of the proposed method.\n3. Comparison with Existing Knowledge Sources: The paper compares the extracted visual knowledge with non-parametric knowledge in existing knowledge graphs and parametric knowledge from large language models. This comparison provides a clear understanding of the unique value proposition of OpenVik.\n4. Real-World Application: The extracted knowledge is integrated across various visual reasoning applications, showing consistent improvements. This indicates the real-world applicability of OpenVik, making it a practical solution for visual knowledge extraction.", "weaknesses": "1. Limited Dataset: The training data for the model is built based on Visual Genome and its relation-enhanced version Dense Relational Captioning. The performance of the model on more diverse datasets remains to be studied, which could limit its generalizability.\n2. Implementation Complexity: The model involves complex implementation details, including the use of a ResNet50-FPN backbone for the open relational region detector and fine-tuning of the visual knowledge extractor. This complexity could make the model difficult to implement and adapt for other researchers or practitioners.\n3. Limited comparisons and lack of ablation studies: The authors only compare to a few baselines, but there are many knowledge-enhanced models which need to be compared in the application section.  \n\nOther detailed questions are listed in Questions section.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***The performance of the model on more diverse datasets remains to be studied. Can this method be scaled to larger training dataset, or other backbones***\n\n**A**: In this work, we chose Visual Genome and its relation-enhanced dataset as the benchmark because they are rich in relational region descriptions, making them the most suitable public data available for training a visual relational knowledge extractor, as far as we know in the existing literature. \n\nIt's essential to clarify, however, that our proposed methods are not constrained to these specific datasets or architectures. They have been designed to be model-agnostic and can be flexibly adapted to other backbones or tested on more diverse datasets. The challenge of gathering such datasets that are capable of supporting deep-semantic understanding and cross-modality grounding is complex. We recognize this as an important avenue for future exploration and are eager to expand our research based on the current work.\n\n>***Implementation complexity***\n\n**A**: Our model is composed of two main components: the region detector and the knowledge generator. Both leverages established backbones. We've tailored these designs specifically for open relational tasks, ensuring we avoid adding undue model complexity. To facilitate ease of implementation and to encourage further research in this domain, we intend to release our comprehensive implementation code along with trained model checkpoints.\n\n>***Need to compare with more other methods for the applications and implementation details***\n\n**A**: In our work, three application tasks were utilized to evaluate the efficacy of OpenVik's generated knowledge by augmenting the input with generated knowledge for the challenging zero-shot settings. It's important to recognize that this augmentation can also be adapted to different backbones or in varied settings, while the optimization for incorporating knowledge across diverse applications is beyond the scope of this work.\n\nTo further showcase generalizability, we have included additional results (Tables 13–15 in the attached PDF) using a contemporary backbone for each application. The results confirm that this new backbone model benefits from OpenVik's contexts, similar to the zero-shot baseline. This finding further substantiates the usefulness of our approach to extracting visual relational knowledge.\n\nWe acknowledge and appreciate your valuable feedback, and we agree that these expanded comparisons and clarifications will strengthen the paper.\n\nFor those seeking more specific details on the application implementations, please refer to the following:\n- Retrieval: lines 276-279 (zero-shot, ZS) and 300-307 (knowledge augmented, knowledge+)\n- GSR: lines 296-299 (ZS) and lines 300-307 (knowledge+)\n- VCR: lines 323-325 (ZS) and 326-329 (knowledge+)\n\n>***Notations and training loss***\n\n**A**: Thanks for pointing out the oversight that needs further clarification. We have listed all notations in Tables 10 and 11 in the attached PDF and will ensure to include these crucial details in the final version. \n\nSpecifically, for the relational region detector, $L_{RD}$ is the regional regression loss from the detector, and $L_{K}$ is the supervision from the ground truth relational knowledge. The training objective $L_l$ of the relational region detector is formulated as $L_v = L_{RD} + L_{K}$. \n\n- **Region Regression**: This part is guided by our created regional box labels, denoted as $U_j$. More precisely, the foreground of these relation-centric region labels is created by taking the union of the object-level bounding boxes of the entities, such as *boat*, *water*, contained in a ground truth region knowledge description $V_j$. This forms the smooth L1 loss $L_{RD}$ [1] for region regression. \n\n- **Knowledge Supervision**: To assist with the refinement of the bounding box, we replaced the object-centric label classification in traditional object detectors with knowledge supervision. A pre-trained generator is finetuned to create the regional description grounded to the given region. This is supervised by the cross entropy loss  $L_K$ with region description $T_j$ [2]. \n\n[1] Ross Girshick. Fast r-cnn. ICCV 2015.\\\n[2] Li et al. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICML, 2022.\n\n>***What is binary mask in the generator***\n\n**A**: Each binary mask represents a detected region from the open region detector, based on which the model would generate region-specific knowledge. \n\n>***Details for in-depth knowledge quality evaluation***\n\n**A**: All the methods compared were based on the same set of 100 randomly selected images to ensure a fair comparison. As detailed in Appendix D lines 561–562, we utilized three different raters for the study. The calculated average pairwise Cohen's κ value is 0.76, demonstrating good agreement.\n"}, {"review_id": "Tnodr8duzB", "reviewer": "Reviewer_GNm1", "summary": "This paper introduces a novel approach to visual knowledge extraction named OpenVik. It comprises three main components: the Open Relational Region Detector, the Format-Free Visual Knowledge Generator, and the Diversity-Driven Data Enhancement module.\n\nThe Open Relational Region Detector, built on the object detection framework FasterRCNN, is fine-tuned to discern relation-oriented visual regions in images. The Format-Free Visual Knowledge Generator's purpose is to utilize this relational region information to create more relation-relevant visual knowledge from an image. The last component, the Diversity-Driven Data Enhancement module, functions as a knowledge post-processing module. Its primary goals include filtering out extraneous knowledge facts and enriching relations via synonym substitution, leveraging external knowledge resources.\n\nThe experimental results indicate that OpenVik surpasses previous benchmarks in most test scenarios, implying its effectiveness in generating high-quality relational knowledge from images.", "questions": "1. With the existence of general multimodal models, why do we need OpenVik?\n\nPlease answer my questions mentioned in the **Weaknesses** .", "limitations": "Good", "rating": 7, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "1. The visual representation of the data through figures and charts is commendable. In particular, Figure 1 effectively illustrates the primary components of OpenVik. The color-coded highlights for 'entities', 'relations', and 'descriptive details' significantly enhance the readability of the examples.\n\n2. The architectural design of the model, along with the selected loss functions, is robust and well-conceived. The three components harmoniously co-function to train on current relational datasets and subsequently generate novel open visual knowledge from images.\n\n3. The level of detail in the explanation of the proposed method is noteworthy. The comprehensive presentation allows readers to gain a thorough understanding of OpenVik even after a single read-through.", "weaknesses": "1. The paper could benefit from a stronger justification for its approach. With the existence of established multimodal models like BLIP[1], mPLUG[2], OFA[3], and recent advancements like BLIP-2[4], MiniGPT4[5], all of which can potentially perform Visual Knowledge Extraction through prompting, the specific need for OpenVik is not explicitly clarified. Further explanation about what unique advantages OpenVik brings would be beneficial.\n\n2. The comparative analysis in section 4.1 appears unbalanced. The use of older models, such as the captioning model and relational captioning model, for benchmarking does not provide a robust basis for comparison. It would be more persuasive if contemporary models were included in the comparison.\n\n3. The ablation study appears to be not comprehensive enough. Crucial distinctions, such as those between the Open Relational Region Detector and the standard FasterRCNN, are not sufficiently elucidated. More detailed examinations of such comparisons would significantly enhance the paper's depth of analysis.\n\n4. The paper lacks clear explanation regarding the evaluation settings, such as the specific split used during evaluation. While I did find these details eventually in the supplementary material, I would recommend the authors incorporate such essential information directly within the main body of the paper. This would ensure a smoother reading experience and easier access to important technical specifications.\n\n[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation\n\n[2] mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections\n\n[3] OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework\n\n[4] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models\n\n[5] MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***What unique advantages OpenVik bring compared with existing multimodal models? With the existence of general multimodal models, why do we need OpenVik***\n\n**A**: Thank you for raising the question regarding the need for OpenVik in the context of existing multimodal models. The specific need for OpenVik lies in addressing key challenges in visual relational knowledge that current models still grapple with:\n\n- **Current Models Exhibit Deficiencies in Compositional Understanding**: Despite the success of large vision and language models (VLMs) in many downstream applications, it is unclear how well they encode compositional information. Recent studies have demonstrated that state-of-the-art VLMs have poor relational understanding and often rely on object-centric shortcuts, leading to the phenomenon that the models behave like a bag of words. This can blunder when linking objects to their attributes, and demonstrate a severe lack of semantic role sensitivity [1, 2]. OpenVik's focus on relational knowledge extraction encourages deep semantic structure understanding, and visual prompting enables semantic grounding across modalities.\n\n- **Relational Knowledge Helps in Complex Planning and Reasoning**: Tasks requiring intricate comprehension, such as planning and reasoning, benefit from relational knowledge [3, 4, 5, 6]. The relational knowledge from OpenVik can enhance compositional generalization in visual reasoning, thereby contributing to the system's trustworthiness and robustness.\n\n- **Openness and Diversity**: OpenVik's specialized approach to open relational detection, format-free knowledge generation, and diversity-driven data enhancement provides unique advantages for discovering nuanced, detailed, verbs that are less seen in the training set. \n\n[1] Yuksekgonul, Mert, et al. \"When and why vision-language models behave like bag-of-words models, and what to do about it?.\" ICLR, 2023.\\\n[2] Hendricks, Lisa Anne, and Aida Nematzadeh. \"Probing image-language transformers for verb understanding.\" ACL, 2021.\\\n[3] Li, Bo, et al. \"Evaluating ChatGPT's Information Extraction Capabilities: An Assessment of Performance, Explainability, Calibration, and Faithfulness.\" arXiv preprint arXiv:2304.11633 (2023).\\\n[4] Wang, Renhao, et al. \"Programmatically Grounded, Compositionally Generalizable Robotic Manipulation.\" ICLR, 2023.\\\n[5] Kurenkov, Andrey, et al. \"Modeling Dynamic Environments with Scene Graph Memory.\" International Conference on Machine Learning. PMLR, 2023.\\\n[6] Hsu, Joy, Jiayuan Mao, and Jiajun Wu. \"DisCo: Improving Compositional Generalization in Visual Reasoning through Distribution Coverage.\" TMLR, 2022.\n\n***It would be more persuasive if contemporary models were included in the comparison***\n\n**A**: We thank the reviewer for the insightful comments. Please refer to our added results under ***Q1*** in the global response. \n\n***Crucial distinctions, such as those between the Open Relational Region Detector and the standard FasterRCNN, are not sufficiently elucidated***\n\n**A**: Please note that a direct comparison between the two may not be straightforward due to their differing objectives. The standard FasterRCNN is designed for object-level detection and classification, wherein the objects are restricted to a predefined set. In contrast, our Open Relational Region Detector is trained to automatically detect relational regions potentially containing relation-oriented knowledge, which is a more complex and nuanced task. \n\nIn response to the comments, we have added an ablation study to illustrate the effects of loading a pre-trained detector backbone versus training the detector from scratch without pre-training. Please refer to our results under ***Q2*** in the global response. \n\n***Incorporate evaluation settings directly within the main body of the paper***\n\n**A**: Thanks for the suggestions! Due to the page limit and the experiment quantity of knowledge generation as well as three downstream tasks, we only included the main module implementation details in the main body, with the specifications for downstream evaluations in Appendix C and D. We appreciate your suggestions on the significance of such details and will ensure that the essential evaluation specifications are included in the main body of the final version. "}, {"review_id": "2Vpm24af7D", "reviewer": "Reviewer_nGQ3", "summary": "Authors present OpenVik, a method for open visual knowledge extraction. It consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest. They explore two data enhancement techniques for diversifying the generated format-free visual knowledge. Extensive knowledge quality evaluations highlight the correctness and uniqueness of the extracted open visual knowledge by OpenVik. Moreover, they show consistent improvements across various visual reasoning applications, indicating the real-world applicability of OpenVik.", "questions": "- How useful are the generated facts?\nThere seems to be a large variance in the entities, relations and the overall details in the facts. When looking at the produced facts, it can be seen that the facts generated for individual image patches could be quite useless and random, e.g. \"dark brown mane growing behind head\", and also not related to the main entities in the image, e.g. zebra.\nThere is no real structure in the facts. \n\n- Do the generated facts suffer from typical problems like hallucination?\nIf this is the case authors should also elaborate on that.\n", "limitations": "The paper should elaborate more on the difficulties and the limitations of their approach.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Well-written\n- Extensive Evaluations (Ablation Study and Applications)\n- OpenVik outperforms traditional approaches and generates rich Scene Graphs\n", "weaknesses": "- The importance of the pre-training for the Open Relational Region Detector is mentioned but not evaluated\n\n- It seems that the performance of OpenVik is mainly based on BLIP's performance\nAblation of OpenVik vs. BLIP is missing.\n\n- Usefulness of Scene Graph needs to be evaluated instead of diversity", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">***The importance of the pre-training for the Open Relational Region Detector is mentioned but not evaluated***\n\n**A**: Thanks for the observation. Please refer to our added results under ***Q2*** in the global response. \n\n>***Ablation of OpenVik vs. BLIP***\n\n**A**: While BLIP primarily focuses on whole image captioning, making it not directly comparable to OpenVik's relational knowledge generation. We take the relational regions detected by our trained region detector as the input for BLIP decoder. Please refer to our updated results under ***Q1*** in the global response. \n\n>***Usefulness of Scene Graph needs to be evaluated instead of diversity***\n\n**A**: Besides the internal knowledge quality evaluation, such as diversity, three downstream external tasks in Section 5 extend beyond these measures. \n\nThese tasks are designed to provide external validation of the knoowledge effectiveness generated by OpenVik and its capability to enhance downstream tasks. We follow a zero-shot setting, while this augmentation can also be conducted with other backbones. Results demonstrate that the backbone model significantly benefits from the context provided by OpenVik, underscoring the value of the extracted visual knowledge.\n\nTo demonstrate generalizability, we expanded our comparisons to include an additional contemporary backbone variant (BLIP, R2C [1]) for each application, as shown below (See full Tables 13-15 in the attached PDF). The results confirm that this added model benefits from OpenVik's contexts, similar to the zero-shot ones.\n\n|  Method  | Recall@1  | Recall@5 | Recall@10 | Avg |\n| -------------------- | ----- | ------- | ------ | -------- | \n| BLIP | 63.11 | 86.30  | 91.10  | 80.17    | \n| OpenVik + BLIP  | 65.23 | 87.71 | 91.90  | 81.61  |\n\n|  Method  | Accuracy  | Precision | Recall | F1 |\n| -------------------- | ----- | ------- | ------ | -------- | \n| BLIP | 70.42 | 65.32 | 69.25 | 67.23  | \n| OpenVik + BLIP  | 80.25 | 72.55 | 70.61  | 71.57  |\n\n Method       | Accuracy | Precision | Recall | F1    |\n| ----------------- | -------- | --------- | ------ | ----- |\n| R2C            | 62.50    | 62.50     | 62.45  | 62.47 |\n| OpenVik + R2C     | 67.40    | 67.54     | 67.43  | 67.48 |\n\n[1] Zellers, Rowan, et al. \"From recognition to cognition: Visual commonsense reasoning.\" CVPR, 2019.\n\n>***How useful are the generated facts? the facts could be quite useless and random, e.g. \"dark brown mane growing behind head\", and also not related to the main entities in the image, e.g. zebra. There is no real structure in the facts.***\n\n**A**: Thank you for your insightful observation regarding the variance in the generated facts. It is essential to understand that the task output (such as \"dark brown mane growing behind head\") is intentionally not following the conventional SVO structure. Our goal is to explore and capture nuanced visual semantics in a more flexible and unconstrained manner, without being restricted by fixed formats. Let's break down the semantics of this example:\n\n- Subject: \"mane\" (referring to the hair on the neck of a mammal like a horse, lion, or zebra)\n- Adjectives: \"dark brown\" (specifying the color of the mane)\n- Verb: \"growing\" (describing the appearance or position of the mane)\n- Prepositional Phrase: \"behind head\" (providing the location of the mane)\n\nThis description offers a detailed and relevant depiction of the main entity, focusing on attributes such as color and position, and does not adhere strictly to the SVO structure.\n\nVisual relational knowledge, as opposed to traditional text-based extraction, emphasizes intricate details like tools, sizes, and spatial relationships. By capturing these complexities, our approach complements existing knowledge literature established on texts, enhancing logical reasoning and promoting explainable AI in visual tasks. This can also alleviate hallucination problems in QA/captioning and add a layer of relational regularization to factual knowledge, mitigating bias in large language model prompting.\n\nWe do recognize that the language generation process may lead to variations in the quality and structure of the facts generated. This is reflective of the diversity and intricacy found within visual information and indeed points to areas for further refinement in our model. However, this does not diminish the significance of open visual knowledge extraction, which serves as an indispensable component in capturing the visual aspect of world information.\n\n>***Do the generated facts suffer from hallucination***\n\n**A**: Thank you for raising the important question regarding potential hallucinations in knowledge generation. It is insightful to recognize that unbalanced and noisy distributions within the training data can indeed lead to errors in the knowledge produced. If we view hallucinations as unwarranted inferences based on input, then such inaccuracies in OpenVik and comparable baselines are more typically a result of detection errors due to data bias by associating wrong features to a class/label. Two illustrative failure cases can be found in the attached PDF, Figure 1. For example, in the left figure, a ladder has been misidentified as a towel, leading to the erroneous description of a 'blue towel hanging from dry shower.' In the right figure, a 'black speaker by flat tv' is generated, although the speaker is not present in the image—possibly reflecting common co-occurrences within the data set. To solve this problem, the key is identifying the cofounder feature of class labeling. \n\nThe unwarranted inference error is more prevalent across QA and captioning tasks, rather than the fact generation process. However, OpenVik is helpful to mitigate the hallucinations in QA and captioning since it can discover unique relational facts. OpenVik's performance metrics demonstrate that it maintains a desirable balance between diversity and validity, offering more accurate grounding that can enhance high-level tasks such as storytelling.\n"}, {"review_id": "1ExGBVwYj7", "reviewer": "Reviewer_QgT9", "summary": "The authors claim that they propose OpenVik which consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest. However, some issues are concerned.", "questions": "1. Paper has weak technical depth, it requires more technical details. \n\n2. Further experimental analysis needed. Please give qualitative validation.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The authors claim that they propose OpenVik which consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest.", "weaknesses": "1. Paper has weak technical depth, it requires more technical details. \n\n2. Further experimental analysis needed. Please give qualitative validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Technical depth and details***\n\n**A**: We would like to stress several distinct technical contributions of OpenVik compared with existing models:\n\n- **Open relational region detector**: Existing detectors often focus on locating objects, while OpenVik is trained to directly detect regions that capture **interactions** and **abstract semantic structures**, such as vivid verbs like “attached to” and nuanced details like “full of”. Some alternative region detectors often need additional visual controls, such as mouse clicks [1, 2] or language controls [3] on a combination of predefined sets of textual properties. OpenVik saves this additional input with its automatic visual grounding ability, which thus improves **knowledge diversity and freshness** significantly. \n\n- **Knowledge generator**: One big difference is that OpenVik is the reasoning-driven generation. Existing regional captioners or knowledge generators often rely on object-level annotation, where the text decoder generates descriptions based on the localized object set [4]. **This leads to the model working like a bag-of-word and a lack of deep semantic understanding.** OpenVik provides **better knowledge grounding** by conditioning the generator on the detected relational region. It includes the ability to **automatically discover the types of interest** that are not only salient but also benefit downstream relational reasoning, as proved in various downstream tasks in Section 5. \n \n- **Language diversity**: Training such a new paradigm for open relational visual knowledge extraction is not trivial. The lack of diverse training data and distribution bias present significant challenges. As shown in Section 4.3 and Figure 3, the **diversity-driven data enhancement** strategies put forth in our work can effectively optimize knowledge richness. They are also generalizable to other models or backbones.\n\nIn summary, our work has attempted to explore uncharted territory in the field, with extensive knowledge evaluation and a variety of downstream tasks demonstrating its effectiveness. \n\n***Further experimental analysis and qualitative validation***\n\n**A**: Thank you for emphasizing the importance of experimental analysis, particularly with respect to qualitative validation. In our study, we have conducted a comprehensive evaluation of the proposed open visual knowledge extraction pipeline. This includes internal assessment of knowledge quality and external evaluation of downstream tasks.\n\n- **Internal Knowledge Quality Evaluation**: We assessed the knowledge generation performance using traditional generative metrics and conducted an in-depth analysis of knowledge quality. Moreover, we compared our approach with existing knowledge sources, as illustrated in Figure 2.\n\n- **External Downstream Tasks Evaluation**: To gauge whether OpenVik's extracted open visual knowledge can enhance reasoning and inference capabilities in various applications, we explored three multimodal tasks, detailed in Section 5. Our analysis included extensive qualitative and quantitative assessments of the results.\n\n- **Qualitative Validation**: We also recognized the importance of providing concrete examples to illustrate our findings. Therefore, Appendix F included a case study featuring qualitative examples of Open Visual Knowledge from OpenVik. More qualitative illustrations from the three downstream applications are available in Appendix G.\n\nBy combining these different methods of evaluation, our study provides a well-rounded perspective on the effectiveness and potential applications of OpenVik. We appreciate the valuable feedback and will ensure that the final version of our paper highlights these aspects in a clear and compelling manner.\n"}]

Author rebuttal:
>***Q1. Adding contemporary regional captioning baselines***

**A**: We appreciate the helpful suggestions on adding region captioning baselines. Note that although the proposed task in our paper has some similarities to region captioning, we would like to highlight the crucial difference in OpenViK: **it is designed to automatically detect regions that can be grounded in diverse and fresh relational knowledge, in contrast with traditional methods where the region detectors primarily focus on object-level areas, or the generators are learned on the bag of words over a fixed class of objects**. 

Following the suggestion from reviewers PUUs and GNm1, we expanded our comparison baselines to include more region captioning methods, including Sub-GC [1], BLIP [2], and BLIP2 [3]. The results are shown below (A full comparison with regional captioning methods is included in Table 12 of the attached PDF). 
|    Method  | BLEU  | ROUGE-L | METEOR | Validity | Conformity | Freshness | Diversity |
| ---------- | ----- | ------- | ------ | -------- | ---------- | --------- | -------- |
| Sub-GC     | 0.272 | 0.263   | 0.221  | 0.892    | 0.871      | 0.795     | 0.547     |
| BLIP       | 0.264 | 0.266   | 0.252  | 0.886    | 0.855      | 0.760     | 0.531     |
| BLIP2      | 0.275 | 0.285   | 0.257  | 0.892    | 0.871      | 0.766     | 0.535     |
| OpenVik  | 0.280 | 0.283   | 0.250  | 0.907    | 0.883      | 0.809     | 0.619     |

It shows that OpenViK has a similar accuracy performance to the recently proposed SOTA approach but is superior in diversity and freshness metrics, which indicates a deep understanding and richer reasoning ability of OpenViK. The comparison also illustrates that OpenVik's specialized designs, including contrastive decoding and knowledge diversity regularizers, confer clear advantages regarding diversity and freshness. 

[1] Zhong, Yiwu, et al. "Comprehensive image captioning via scene graph decomposition." ECCV, 2020. \
[2] Li, Junnan, et al. "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation." International Conference on Machine Learning. PMLR, 2022. \
[3] Li, Junnan, et al. "Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models." arXiv preprint arXiv:2301.12597 (2023).

>***Q2. Ablation on the pre-training for the open relational region detector***

**A**: To address this concern, we have conducted an additional ablation study, contrasting the outcomes when loading a pre-trained detector backbone with training the detector from scratch (The full ablation can be referred to in Figure 13 of the attached PDF):

|  Variant | BLEU  | ROUGE-L | METEOR | Validity | Conformity | Freshness | Diversity |
| -------------------- | ----- | ------- | ------ | -------- | ---------- | --------- | --------- |
| w/o PreDet | 0.201 | 0.275   | 0.230  | 0.812    | 0.833      | 0.701     | 0.502     |
| Full Model           | 0.280 | 0.283   | 0.250  | 0.907    | 0.883      | 0.809     | 0.619     |

The ablation results show that omitting the pre-training step of the FasterRCNN model tends to result in the detection of more overlapping regions. This, in turn, causes a noticeable decrease in both knowledge diversity and freshness in the detected regions, which indicates the importance of loading the pre-trained model for region detection. 


Meta-review: This paper introduces a new paradigm of open visual knowledge extraction called OpenVik. The proposed framework generates format-free knowledge by prompting a large multimodality model with detected regions of interest. It consists of an open relational region detector and a format-free visual knowledge generator. The paper highlights the limitations of existing approaches to visual knowledge extraction and demonstrates the correctness and uniqueness of the extracted open visual knowledge by OpenVik. The extracted knowledge is integrated across various visual reasoning applications, showing consistent improvements and indicating the real-world applicability of OpenVik.

Strengths:
1. Significance: open visual knowledge extraction is an attractive goal, which extracts knowledge from unlabeled image data.
2. Novelty: The proposed framework is a novel in the field.
3. Evaluation: solid evaluation of the generated knowledge, using traditional generative metrics and in-depth knowledge quality assessment. evaluation on various downstream tasks.
4. Analysis: the comparison of the extracted knowledge with non-parametric knowledge in existing KG and parametric knowledge from LLMs illustrates the unique value proposition of OpenVik.

Weaknesses:
1. The reviewers also have concerns over the complexity of the approach
2. Lack of ablation studies (e.g. knowledge-enhanced models, FasterRCNN).
3. The reviewers also like to see an important set of baselines involving multimodal models.
Decision: Accept (poster)