Title: Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting
Venue: NeurIPS 2023
Accepted: yes
Keywords: Visual Knowledge Extraction

Multimodality

Large Model Prompting
Abstract: Images contain rich relational knowledge that can help machines understand the world. Existing methods on visual knowledge extraction often rely on the pre-defined format (e.g., sub-verb-obj tuples) or vocabulary (e.g., relation types), restricting the expressiveness of the extracted knowledge. In this work, we take a first exploration to a new paradigm of open visual knowledge extraction. To achieve this, we present OpenVik which consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest. We also explore two data enhancement techniques for diversifying the generated format-free visual knowledge. Extensive knowledge quality evaluations highlight the correctness and uniqueness of the extracted open visual knowledge by OpenVik. Moreover, integrating our extracted knowledge across various visual reasoning applications shows consistent improvements, indicating the real-world applicability of OpenVik.

Open Visual Knowledge Extraction via Relation-Oriented Multimodality Model Prompting

Abstract

Images contain rich relational knowledge that can help machines understand the world. Existing methods on visual knowledge extraction often rely on the predefined format (e.g., sub-verb-obj tuples) or vocabulary (e.g., relation types), restricting the expressiveness of the extracted knowledge. In this work, we take a first exploration to a new paradigm of open visual knowledge extraction. To achieve this, we present OpenVik which consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest. We also explore two data enhancement techniques for diversifying the generated format-free visual knowledge. Extensive knowledge quality evaluations highlight the correctness and uniqueness of the extracted open visual knowledge by OpenVik . Moreover, integrating our extracted knowledge across various visual reasoning applications shows consistent improvements, indicating the real-world applicability of OpenVik .

1 Introduction

Knowledge extraction has been widely studied on texts [8, 1, 13, 9] for enhancing logical reasoning [45, 14, 6] and explainable AI [18, 57, 5, 55], and recent studies have explored open knowledge extraction through categorizing seed relations [64, 40] and eliciting from language models [47]. Visual knowledge extraction, on the other hand, captures intricate details like tools, sizes, and positional relationships, which are often difficult to express exhaustively in texts [39, 28, 48, 7]. Yet existing approaches of visual knowledge extraction are either restricted by a fixed knowledge format [52, 63, 20, 22] or the predefined sets of objects/relations [52, 63, 21]. While efficient at capturing interactions between objects, the produced visual knowledge is often limited in richness and confined to a single format, falling short in representing the diverse real-world information that can be complemented by visual data.

In this endeavor, we propose to further explore a new paradigm of open visual knowledge extraction ( OpenVik ). Specifically, we propose to generate relation-oriented, but format-free knowledge that includes a wider variety of elements, such as descriptions, insertions, and attributes, among others. Drawing inspiration from the wealth of knowledge encapsulated in large models [49, 61, 46], we propose to leverage pre-trained large multimodality models by eliciting open visual knowledge through relation-oriented visual prompting. This approach allows for a more nuanced understanding of visual data, mirroring how humans naturally emphasize certain aspects of visual scenes when perceiving and describing visual information, leading to more flexible visual knowledge extraction.

Our proposed OpenVik framework consists of two modules, an open relational region detector and a format-free visual knowledge generator. It is a unique challenge to detect the regions potentially containing relational knowledge, since traditional region detectors primarily focus on learning predefined object classes. To learn the regression of relational regions, we propose to use free-form knowledge descriptions as supervision and leverage knowledge generation as a training objective. With the detected regions, the remaining question is how to interpret these regions into free-form knowledge. We propose a visual knowledge generator by harnessing the power of language variety enhancement in large pre-trained multimodality models. Specifically, we prompt them to generate knowledge descriptions of any formats and condition the generation on the detected relational regions.

However, establishing a new paradigm of open visual knowledge extraction is challenging due to the absence of comprehensive and diverse training data. Existing datasets sources such as scene graphs [51, 24], dense captions [20], and dense relational subsets [22] often exhibit a long-tail distribution biased to more prevalent relations and entities [44]. Brute-force merging of these datasets could exacerbate the distribution bias inherent in the data. To alleviate the bias, we propose two diversity-driven data enhancement strategies based on an adapted TF-IDF+ score, involving random dropping and data augmentation with external knowledge resources. These strategies optimize data distributions and richness, thus fostering diverse open visual knowledge extraction.

We implement extensive evaluations to assess the quality and utility of the open visual knowledge extracted by OpenVik , encompassing: 1) directly evaluating the performance of knowledge generation; 2) engaging human evaluators for a multi-faceted assessment of in-depth knowledge quality; and 3) comparing the open visual knowledge extracted with OpenVik with existing knowledge sources, such as non-parametric knowledge from the ConceptNet knowledge graph, and parametric knowledge from the GPT-3.5 large language model. Furthermore, the utility of the extracted open visual knowledge is validated through its integration with several common applications that require visual understanding, including text-to-image retrieval, grounded situation recognition, and visual commonsense reasoning. These applications demonstrate consistent improvements, affirming the practical utility of OpenVik .

2 Related Work

Visual knowledge extraction. Recent advancements in knowledge extraction have extended from being purely text-driven to incorporating images [11, 29]. VisKE [39] is designed to verify relations between pairs of entities, e.g., eat ( horse , hay ). Scene graphs, which locate objects in the image and identify visual predicates between subjects and objects in a triple format, e.g., ( man, on, chair ), are extensively studied for vision understanding [52, 63, 60]. A recent work OpenSGG [17] extends SGG to open-vocabulary objects, enabling the relation prediction for unseen objects. Other studies have explored caption-like formats, like dense captioning [20] with a set of object-centric descriptions across regions, and relational captioning [22] focusing on relational information between objects. Despite these advancements, existing methods either adhere to a pre-defined format and vocabulary or are constrained by the biased distribution of training sets. This highlights the pressing need for a format-free approach in visual knowledge extraction with knowledge diversity.

Large model prompting. Recently, large language and multimodality models have exhibited remarkable successes in capturing commonsense knowledge across various tasks, especially facilitating few-shot [15, 53, 25, 58] and zero-shot learning [23, 66, 59]. The potential of prompt-based learning for pre-trained vision-language models [2, 37, 42] has been explored for handling diverse data types across multiple modalities, such as images and texts, with improved performance in tasks including image classification [33, 67], segmentation [32] and visual question answering [16]. Leveraging the substantial information encapsulated within these pre-trained multimodality models to extract explicit knowledge can enrich existing resources, potentially laying the groundwork for advances in interpretability research and mitigating the hallucination issue associated with large models [19, 10].

3 Method

In this section, we introduce our new paradigm and two key model design novelty featuring OpenVik , relation-oriented multimodality model prompting and diversity-driven data enhancement.

+Relation

a boat rests on water

+Entity

the jet in blue sky

smoke path in air

relation enhanced

entity enhanced

Detector

Training Data

the boat on water

plane in air

crowd stand on ground

Diversity-Driven Data Enhancement

Format-free Visual Knowledge Generator

Open Relational Region Detector

Random Dropping

Augmentation with External

Resources

Inference

Relation-Oriented Regions

𝓛

𝑲

selected original data

Low

𝓛

𝑹𝑫

𝓛

𝑴𝑳𝑬

Open Visual Knowledge

OpenVik

Large Multimodality

Model

TF-IDF+

High

𝓛

𝑽

Generated Knowledge

Similarity

KG

LLM

Prompting

large boat docked at pier

flying jet leaving behind smoke

building blocks across from brave blue water

standing crowd watch as a jet flies high above water

3.1 Open Visual Knowledge Extraction

Given a dataset D = { ( I i , T i , U i ) } M i =1 consisting of M samples, I i is the i -th image (such as the input image in Figure 1), T i = {T j } n i j =1 is a set of n i region descriptions (such as ' the boat on water ' in Figure 1), U i = {U j } n i j =1 is the set of n i relation-oriented visual regions, where each T j corresponds to a visual region U j ∈ U i in image I i . The goal of our open visual knowledge discovery is to train a model M capable of producing a set of format-free knowledge descriptions (such as ' large boat docked at pier ' in Figure 1) given any image I k during the inference stage.

3.2 Relation-Oriented Multimodality Model Prompting

The overall architecture of OpenVik is shown in Figure 1. It comprises two modules: an open relational region detector M v and a format-free visual knowledge generator M t . The two modules are learned separately during training with our diversity-enhanced data (Section 3.3) and combined to produce format-free visual knowledge at inference. Specifically, the relational region detector M v takes an image I i as the input and learns to select a flexible number of relational regions U i = { ( U j ) } n i j =1 that captures object interactions, each corresponding to a description T j in T i ; the visual knowledge generator M t generates format-free knowledge descriptions by prompting and fine-tuning the multimodality model with the guidance of detected visual region U j . All notations for the region detector and knowledge generator are detailed in Table 9 and Table 10, respectively.

Open relational region detector. Although existing object detection algorithms have been widely recognized for their efficiency in object detection, they are usually restricted to object-centric visual regions in a predefined set, and thus cannot directly capture open relational information with a single box. Detecting regions containing relational knowledge remains to be a challenge. We make two adaptions on the object detection FasterRCNN [38] to train the open relational region detector:

Region Regression : we change the original object-centric region labels to our newly created relationcentric box labels, denoted as U j . The foreground of each relation-centric region label U j is created by taking the union of the object-level bounding boxes of the entities, i.e., boat, water , contained in a ground truth region knowledge description T j . This forms the region regression loss L RD.

Knowledge Supervision : To assist with the refinement of the bounding box, we replaced the object-centric label classification in traditional object detectors with knowledge supervision. A pre-trained generator is finetuned to create the regional description grounded to the given region. This is supervised by the cross-entropy loss L K with region description T j .

The training objective L l of the relational region detector is formulated as below, where L RD is the regional regression loss and L K is the knowledge supervision loss,

Format-free visual knowledge generator. OpenVik provides better knowledge grounding by conditioning the generator on the detected relational region, leading to a reasoning-driven generation. Specifically, the detected bounding box (such as the box containing ' boat ' and ' pier ' on the far left) is utilized as a visual prompt when fine-tuning the visual knowledge generator. The model architecture of the knowledge generator is built upon a combined large multimodality model, which composes a pre-trained vision transformer ViT-B [12] and the image-grounded text decoder of BLIP [27]. The two modules are jointly trained on a generic image-text paired dataset comprising over 14 million entries and fine-tuned on the image captioning task, which delivered state-of-the-art performance.

In our visual knowledge generator, the decoder takes the ViT visual representation of the entire image as input and leverages the detected regional mask as a binary visual prompt. This prompt aids in filtering out the background and directing attention toward the relational foreground. The generation of format-free knowledge from the decoder is supervised by the language modeling loss L MLE, which further refines visual attention during the knowledge generation process. As a result, our approach facilitates the production of format-free outcomes that extend beyond the conventional sub-verb-obj form. Besides, to improve information variety, we introduce an amplifying penalty factor for highly similar knowledge generation. For any two generated sequences T a and T b describing image I i ,

where N i is the number of generated knowledge of image I i , s ( T a , T b ) indicates the semantic cosine similarity, and ϕ is a hyper-parameter set as 0.01 controlling the penalty on sequences with only slight difference (e.g. ' dog chasing the man ' and ' dog licking the man ') to be relatively small.

The training objective L l of the format-free visual knowledge generator is formulated as

where α is a weighting hyper-parameter we set as 0.7. The trained relational region detector and visual knowledge generator are combined during inference. Given any image I , the open relational region detector first detects a flexible number of open relations regions of interest, then each detected region R is passed to the format-free visual knowledge generator, where a relation-oriented format-free knowledge phrase (such as ' flying jet leaving behind smoke ' in Figure 1) is generated to describe the given visual focus subarea R of the image. To further encourage within-sequence language variety during inference, we leverage the contrastive decoding strategy from [43], which improves over nucleus sampling and beam search.

3.3 Diversity-driven Data Enhancement

The training data for relational knowledge extraction usually exhibits a long-tail distribution, where more prevalent but simple relations such as in , on , and wear dominate the training set [44]. Consequently, the model trained with such a biased dataset may render limited, and repetitive knowledge. As a remedy, we propose two data enhancement techniques to optimize the data distribution. As the foundational measure for given relation r 's importance, we design a grid TF-IDF+ score S r [34, 54]:

where N is the total number of knowledge phrases in the datasets, f r is the number of occurrences of the relation r , α 1 and α 2 are the grid scales whose values are selected based on f r .

Random dropping on low-quality data. We first remove repeated knowledge descriptions in the same image and then randomly drop descriptions that contain frequently occurring yet meaningless

relations with a low S r (e.g., ' people on ground ') from the original dataset. Specifically, if the S r of the relation in a description is relatively low, i.e., 0.4, we remove it at a random dropping rate of 0.5. This process repeats for all descriptions in an image until the remaining set is 0.6 times the size of the original training set. Consequently, the training data bias is mitigated by removing low-quality data.

Data augmentation with external knowledge resources. For the relations with high TF-IDF+ scores, we leverage external knowledge resources from both non-parametric (i.e., ConceptNet [41]) and parametric (i.e., COMET [4]) knowledge resources to promote diverse knowledge generation [56]. ✔ Enhance Relation Recognition : For each training description with a high TF-IDF+ score, we perform semantic parsing to get all the objects and complement additional relations (e.g., ' rest ' in Figure 1) between each pair of them by mapping the nodes and retrieving edges from the ConceptNet. Each retrieved knowledge triplet is converted to a knowledge phrase and added to the training set for generator training. With this introduced external knowledge, the knowledge generator ultimately yields a more robust and detailed representation of the underlying visual information of objects. This, in turn, bolsters the relation recognition of the visual knowledge generator. ✔ Boost Entity Perception : For the description with the highest-scored TF-IDF+ relation given each image, we also leverage ConceptNet to enrich similar objects (e.g., ' jet ') to the original object (e.g., ' plane '). Additionally, we further introduce new entities (e.g., ' smoke ' in Figure 1) and attribute descriptions (e.g., ' blue ') by prompting the pre-trained attribute commonsense branch of the COMET model (Refer to Appendix A for more details). The entity-based enrichment potentially helps in boosting entity understanding and at the same time enhances the occurrence of important but rare relations in the training set.

3.4 Implementation Details

Our training data are built based on Visual Genome [24] and its relation-enhanced version Dense Relational Captioning [22]. Each sample includes an image identified by a unique ID and a set of relational descriptors describing interactions among objects in the image. Specifically, each relational descriptor includes the full description text, the subject and object names contained in the description text, the relation between them, as well as the bounding box coordinates of the subject and object. The dataset statistic information is summarized in Table 8 in the Appendix B.

Our model is implemented in PyTorch [35] and trained on two Quadro RTX 8000 GPUs. The open relational region detector is initialized from the ResNet50-FPN backbone, then finetuned for another 20 epochs with the relational bounding box. The model detects a maximum of 30 bounding boxes for each image with the highest confidence to avoid misleading noises. The format-free visual knowledge generator is initialized from BLIPbase with the basic ViT-B/16 and finetuned for 20 epochs. Full details on learning parameters can be referred to in Appendix C.

4 Evaluation

In this section, we directly evaluate the extracted open visual knowledge from OpenVik from two perspectives: (1) knowledge generation performance with traditional generative metrics and in-depth knowledge quality assessment; (2) comparison with existing knowledge sources. Besides, ablation studies are conducted to study the influence of diversity design on the generated knowledge and data.

4.1 Evaluation on Generated Knowledge

Generation performance. To directly evaluate the visual knowledge generator, we compare the knowledge generated by OpenVik with a variety of baselines, including scene graph generation [52, 63, 44, 17] (of which Ov-SGG employs an open vocabulary), dense relational captioning [22], and region captioning [20, 65, 27, 26]. Evaluation metrics are traditional language generation measures such as BLEU, ROUGE-L, and METEOR. The results, displayed in the left side of Table 3, reveal that OpenVik outperforms captioning-based approaches and yields results on par with the best scene graph generation baseline. These findings underscore the effectiveness of the format-free visual knowledge generator through relation-oriented prompting of the large multimodality model.

In-depth knowledge quality. To more thoroughly evaluate the quality and richness of the formatfree visual knowledge extraction, beyond simply evaluating it as a language generation model with the

limitation of training data, we incorporate four additional metrics [31], which delve into an in-depth quality evaluation of the extracted visual knowledge from four distinct perspectives:

Validity ( ↑ ): whether the generated visual knowledge is valid to human.

Conformity ( ↑ ): whether the generated knowledge faithfully depicts the scenarios in the images.

Freshness ( ↑ ): the novelty of the knowledge, i.e., the proportion not present in the training set.

Diversity ( ↑ ): the language variance between a randomly sampled pair of knowledge pieces.

Among the four metrics, both the validity and conformity metrics involve human annotators. We randomly selected 100 images as the evaluative subset. Details regarding the scoring guidance and the interface provided to the annotators can be found in Appendix D. The remaining metrics, i.e., freshness and diversity, are calculated automatically. The in-depth knowledge quality evaluation results are displayed in the right part of Table 3, where the average pairwise Cohen's κ on human evaluation results is 0.76 (good agreement). The findings demonstrate that trained with the diversityenhanced datasets, the format-free visual knowledge extracted by OpenVik significantly outperforms other types of baselines in terms of all four metrics. The improvement of diversity, in particular, reaches 14% relatively compared with the inference results from the second runner DenseCap, indicating the advantage of OpenVik in generating rich and comprehensive visual knowledge.

4.2 Comparison with Existing Knowledge Sources

Motion Dynamics,

Attributes,

Belongings

Non-parametric

Knowledge

Parametric

Knowledge

OpenViK

Descriptive

Elements

Abstract Insights

Logical Relations

Examples

Knowledge Source

dog IsA animal (ConceptNet); conceptnet IsA knowledge graph (ConceptNet);

Non-parametric Knowledge

dog HasProperty black (ConceptNet); dog and brown fur covering black (OpenVik)

computer HasA keyboard (ConceptNet); keyboard with computer (OpenVik)

people using light bulbs to illuminate the room; (LLM)

Parametric Knowledge

yellow sign in corner (both); black seat attached to bike (both);

three layer cake on table; blue trash can full of garbage next to brown dresser; blue

box sitting beside a sneaky garage; (OpenVik)

Open Visual Knowledge

(OpenViK)

people wearing fashionable black hats are skiing; baby elephants walking around

adventurous wood; (OpenVik)

the light shining from bright black background; hanging fan are above tall shelf;

brown chair in the background of the room; (OpenVik)

Figure 2: The Venn diagram of knowledge comparison between the open visual knowledge from OpenVik with the non-parametric knowledge from existing knowledge graph (i.e., ConceptNet) and parametric knowledge from large language model (i.e., COMET).

We compare the extracted visual knowledge with the non-parametric knowledge in the existing knowledge graph (KG) and the parametric knowledge from the large language model (LLM). The comparison insights from the three knowledge resources are shown in the Venn Diagram in Figure 2.

Compare with non-parametric knowledge. We take ConceptNet [41] as the representative in the comparison with non-parametric knowledge. To map the knowledge generated by OpenVik to ConceptNet, we parse the knowledge into triplets and associate the endpoints of these triplets with

nodes in ConceptNet. Then we calculate the similarity of embeddings 3 between the parsed relation and all the edge relations among the mapped nodes in ConceptNet. If the similarity score exceeds a predetermined threshold, i.e., 0.75, we consider the mapping successful. As illustrated in Figure 2, we observe that compared with the non-parametric knowledge in KG, the extracted visual knowledge captures richer and more meaningful spatial details, e.g., ' three layer cake on table ', and motion dynamics, e.g., ' baby elephants walking around adventurous wood '.

Compare with parametric knowledge. We compare with parametric knowledge contained in LLM by prompting the gpt-3.5-turbo 4 model with the object information in the image. The prompt template used is detailed in Appendix E. The mapping process follows the approach mentioned earlier. It is found that compared with the parametric knowledge in LLM, the extracted visual knowledge exhibits unique fine-grained visual details, e.g., ' red sticker on fence ', and provides precise scene information, e.g., ' the light shining from bright black background '.

4.3 Ablation Study

The influence on knowledge quality with information variety regularization and data strategies. We conducted ablation studies to evaluate the effectiveness of the information variety regularizer, L V, and our diversitydriven data enhancement strategies. This involves an in-depth assessment of knowledge quality on the same eval-





























































v





aug



aug



uation subset. The results are presented in Figure 3. It is evident from the results that our proposed information variety design primarily impacts freshness and diversity, without compromising validity and conformity. For the freshness, the omission of data augmentation for entities and relations results in the most significant performance degradation. This implies the crucial role these strategies play in infusing novel knowledge into the generation process. As for diversity, the most notable changes in metrics are observed when the L V and random dropping are removed. The strategy for augmenting entities and relations also plays a valuable role in enriching diversity.

Ablation of the pre-training for the open relational region detector. We conducted a comparison of the outcomes when loading a pre-trained detector backbone versus training the detector from scratch, as shown by the yellow bar in Figure 3. Results demonstrate a noticeable decrease in both knowledge diversity and freshness, which indicates the importance of loading the pre-trained model for region detection. This may be because omitting the pre-training step of the FasterRCNN model tends to result in the detection of more overlapping regions, which in turn causes the drop.

The influence on dataset diversity with data strategies. We conduct a direct analysis of the knowledge diversity of the existing datasets and our diversity-enhanced one, compared with the visual knowledge generated from OpenVik . The findings, presented in Table 4, show that the diversity-driven data enhancement strategies significantly boost knowledge diversity. Trained with this enhanced data, OpenVik can extract visual knowledge that exhibits greater diversity than that found in the Visual Genome and Relational Caps , indicating the advantage of OpenVik to format-free visual knowledge generation and its ability to yield richer knowledge diversity.

4.4 Case Study

We present two case studies in Figure 4 (See Appendix F for more) to showcase the format-free visual knowledge generated by OpenVik , in comparison to Visual Genome (Scene Graph and Region Description) and Relational Caps. Contrary to the rigidity of scene graphs, which strictly adhere to a

predefined format, OpenVik can generate knowledge with a flexible semantic structure, not strictly bound to the sub-verb-obj format (e.g., ' blue post attached to wall with white letter '). Examples of this adaptability are highlighted in red. When compared to dense region descriptions, the relational knowledge extracted by OpenVik offers a deeper understanding of the multiple entity interactions within an image. In comparison to Relational Caps, which mainly focus on interactions between two objects, OpenVik significantly broadens the diversity of relation with vivid verbs (e.g., ' attached to ', ' adorning '). Moreover, it introduces novel entities (e.g., ' post ', ' mane ') and enriches the knowledge representation with nuanced details (e.g., ' full of ', ' striped ') that are missed by Relational Caps.

Visual Genome-Scene Graph:

<drink, in, cooler>

<orange, in, box>

<banner, on, building>

<item, on, table>

Visual Genome-Region Descriptions:

oranges in a wood thing

green leaves on oranges

red writing on a white sign

drink in red cooler

Relational Caps:

snow-covered oranges in wood thing

the frost on snow-covered oranges

green leaves on snow-covered oranges

red writing on white sign

OpenVik

:

blue post attached to wall with white letter

the open window to snowy ground

wood box full of different size of orange

white banner on a building with letter o

blue box sitting beside a sneaky garage

a orange covered with ice and green leaves

Visual Genome-Scene Graph:

<hair, on, head>

<zebra, eat, grass>

<eye, on, zebra>

<grass, on, ground>

Visual Genome-Region Descriptions:

black and white striped leg

light shining on the zebra

thin line of black hair

two zebras grazing in the grass

OpenVik

:

striped mane belongs to grazing zebra

zebra with striped ears eating green grass

white stripe adorning leg

dark brown mane growing behind head

grass everywhere surround standing zebra

black nose above green lively grass

Relational Caps:

sticking up ear of grazing zebra

black eye of eating zebra

grazing zebra in green grass

the muzzle of grazing zebra

ORA

C.AY

10X

F/AN

GAI

MAI

SU

Note that we observe the unbalanced and noisy distributions within the training data can lead to errors in the knowledge produced. Viewing hallucinations as erroneous inferences based on input, the inaccuracies observed in OpenVik and similar baselines often stem from detection errors. These errors are typically caused by data biases that incorrectly associate features with a specific class or label. We further two illustrative failure cases in Figure 5. For example, a ' black speaker by flat tv ' is generated, although the speaker is not present in the image-possibly reflecting common co-occurrences within the dataset. Similarly, a ladder in the right figure has been misidentified as a towel, leading to the erroneous description of a ' blue towel hanging from dry shower '. The key to mitigating such incorrect inference is identifying the cofounder feature of class labeling.

Visual Genome-Scene Graph:

<window, in, screen>

<eye, of, cat>

<keyboard, on, laptop>

<cat, beside, laptop>

Visual Genome-Region Descriptions:

laptop key board

cat beside the laptop

pictures on wall behind cat

the cat is brown

Relational Caps: white cat beside black laptop the window in screen the wall behind cat

OpenVik

:

white cat sitting near computer

black speaker by flat tv

black wire plugged into computer

the picture in screen

OpenVik

:

framed picture hanging on wall

recessed lighting in ceiling

blue towel hanging from dry shower

the sunlight coming through window

Visual Genome-Scene Graph:

<paint, in, frame>

<picture, on, wall>

<window, in, bathroom>

<curtain, over, window>

Visual Genome-Region Descriptions: leg of an elephant baby elephant in grass Grass within the enclosure A big rock on the ground

Relational Caps:

blue painting in metal frame the shampoo bottle on rack blue jar in window

5 Application

This section explores whether the extracted open visual knowledge from OpenVik can bolster reasoning and inference capabilities in multimodality downstream tasks by augmenting a baseline in the challenging zero-shot setting.

5.1 Text-to-Image Retrieval

Task Setting. In the text-to-image retrieval task, a given caption is matched to a large set of candidate images, with the most relevant image returned as the result. Adopting the challenging zero-shot

setting, we generate the visual representation v and textual representation t of the given image I and caption T using a pre-trained clip-retrieval model [3]. The baseline involves the image and text embedding similarly based on zero-shot CLIP Retrieval [3] and the fine-tuned model from BLIP [26].

To explore the potential of the extracted visual knowledge from OpenVik , we enrich the given caption T with related contexts derived from the extracted visual knowledge. Specifically, for each query caption, we parse the caption to extract all subject-object pairs ( s, o ) with the NLTK parser. Then s and o are mapped to the open visual knowledge, where knowledge phrases that contain relations occurring more than 30% of the time between s and o are enriched to the original caption T .

Original text: A little room and dining room area with furniture. A living room with a big table next to a book shelf. A living room decorated with a modern theme. A living room with wooden floors and furniture. The large room has a wooden table with chairs and a couch.

Enriched text: big table in room. a decorated living room with wooden furniture. brown couch in room. book on table. wooden table with shelf. shelf next to couch. wooden bookshelf with books next to table.

Qualitative examples. Figure 6 presents an example of OpenVik -based visual knowledge enrichment on captions. By incorporating related contexts from the generated open visual knowledge, the enriched captions convey more precise visual details, which enhances the alignment for text-image alignment.

Quantitative results. Wecurated a subset of 680 images from the testing set of the MS-COCO dataset containing parsed knowledge with at least eight nouns. This ensures an adequate degree of enrichment is achieved through the use of OpenVik . Standard image retrieval metrics, i.e., Recall@1/5/10/ and Avg , are employed to evaluate the performance. The results are presented in Table 5. It is evident that relational context enrichment leads to the average correction of more than 6.0% of the initial zero-shot, highlighting the practical benefits of extracted visual knowledge in visual reasoning tasks.

5.2 Grounded Situation Recognition

Task setting. The event type prediction for the grounded situation recognition task is to predict the best match from predefined 504 event types [36] based on the image. We convert each candidate event verb into a description T : ' An image of <verb> ' for image description matching. Similarly to text-to-image retrieval, we include zero-shot CLIP and the fine-tuned model from BLIP as baselines.

To enrich with contextual knowledge from OpenVik , for each given verb v , we find its nearest synonym in the extracted open visual knowledge and enrich the text description with the most common knowledge phrase containing it, regularized by the objects present in the image. Instead of directly concatenating the retrieved knowledge triplets to the original textual description, we employ an additive decomposition strategy: the similarity s ( I , v ) of the candidate verb v with respect to the given image I is calculated as s ( I , v ) = 1 | D ( v ) | ∑ d ∈ D ( v ) ϕ ( I , v ) , where D ( v ) is the set of descriptors, including the original description and the enriched ones, and ϕ represents the single log probability that descriptor d pertains to the image I .

Verb:

talking

Original text: This is an image of talking. Enriched text: man talking on a small white telephone. adult male with white shirt talking on chatty cellphone. the man waving arms is talking on phone.

Qualitative examples. Figure 7 presents a qualitative example of OpenVik -based context enrichment in the grounded situation recognition task. We observed that verbs like 'shopping' and 'talking' were appropriately enriched with their frequently occurring contexts from the open visual knowledge, leading to a reduced embedding distance between the description and its matching image.

Quantitative results. We assembled a test set of 900 samples from the testing set of GSR that included verbs such as 'talking', 'filming', and 'picking', among others, from a list of 256 words

that can be accurately mapped to extracted visual knowledge, as well as 138 verbs that have a fuzzy match through ConceptNet embedding comparison. The full lists of the exact and fuzzy-matched verbs are detailed in Appendix H. The evaluated metrics include Accuracy, Precision, Recall, and F 1 . The results are presented in Table 6. It can be observed that knowledge enrichment significantly outperforms the zero-shot and BLIP baselines. This suggests that the verb-related contexts introduced by OpenVik -generated knowledge are intuitive and greatly assist in understanding the semantics of event verbs, bolstered by related visual information.

5.3 Visual Commonsense Reasoning

Task setting. The goal of visual commonsense reasoning is to predict an answer from four given option candidates for a given image and question. For the baseline approach, we compare the backbone model R2C from the VCR paper [62] and BLIP [27]. In the visual knowledge-enhanced OpenVik Enriched approach, we perform two-level context augmentation, incorporating both entities and relations: (1) we parse the question and options to obtain all (S, O) pairs and, for each entity pair, apply the same relation augmentation as in the image retrieval task; (2) for the V in each option, we enrich the visual context using the same method as illustrated in grounded situation recognition.

Question

:  Is  Person1 winning the game? the

person engaged in a game.

A

Yes, he is about to go for a run. the person

engaged in a game.

the person walking near runway.

B

No, he is losing. the person engaged in a game.

frustrated person lose game.

C

No, he' s not really enjoying it. the person

engaged in a game. person enjoy at celebration.

D

Yes, he looks like he has a good hand. the person

engaged in a game. the person is watching left hand.

Answer: D

Yes, he looks like he has a good hand.

Qualitative examples. Figure 8 presents an example before and after applying the two-level visual knowledge-based enrichment for visual commonsense reasoning. The results indicate that visual knowledge enhances the correspondence between the correct answer and the image itself.

Quantitative results. We assembled a test set of 939 samples from the validation set of the VCR dataset [62]. Each sample in this test set contains questions and answers with a minimum of five nouns and two relations, guaranteeing an adequate level of information complexity for meaningful engagement with open visual knowledge. The results can be found in Table 7. We observe that the enriched visual knowledge helps especially when solving reasoning questions on humans and their interactions with visually impressive entities, such as ' game ' in Figure 8. This enhancement results in a performance improvement above 3.0% over the zero-shot baseline.

6 Conclusion, Limitations, and Future Work

This work is the first exploration of a new paradigm of open visual knowledge extraction, which combines an open relational region detector to flexibly pinpoint relational regions and a format-free visual knowledge generator that generates visual knowledge by prompting a multimodality model conditioned on the region of interest. To further enhance the diversity of the generated knowledge, we explore two distinct data enhancement techniques. Extensive knowledge evaluations underscore the correctness and uniqueness of our extracted open visual knowledge, and the consistent improvements observed across various visual reasoning tasks highlight the real-world applicability of OpenVik .

While our approach has been shown effective in various scenarios, its performance at larger scales or on more diverse datasets remains to be studied. Future work could investigate its effectiveness across a broader range of tasks and contexts. Also, the current model requires fine-tuning for the visual knowledge extractor. Developing a model that can generalize well with prompt tuning or demonstration augmentation could be another interesting direction for future work.

7 Acknowledgments

Carl Yang was supported by the National Institute Of Diabetes And Digestive And Kidney Diseases of the National Institutes of Health under Award Number K25DK135913.

References

Harith Alani, Sanghee Kim, David E Millard, Mark J Weal, Wendy Hall, Paul H Lewis, and Nigel R Shadbolt. Automatic ontology-based knowledge extraction from web documents. IEEE Intell Syst , 18:14-21.

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NeurIPS , 2022.

Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them. https://github.com/rom1504/clip-retrieval , 2022.

Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, and Yejin Choi. COMET: commonsense transformers for automatic knowledge graph construction. In ACL , 2019.

Aaron Chan, Jiashu Xu, Boyuan Long, Soumya Sanyal, Tanishq Gupta, and Xiang Ren. Salkg: Learning from knowledge graph explanations for commonsense reasoning. In NeurIPS , 2021.

Xuelu Chen, Ziniu Hu, and Yizhou Sun. Fuzzy logic based logical query answering on knowledge graphs. In AAAI , 2022.

Hejie Cui, Rongmei Lin, Nasser Zalmout, Chenwei Zhang, Jingbo Shang, Carl Yang, and Xian Li. PV2TEA: Patching visual modality to textual-established information extraction. In ACL , 2023.

Hejie Cui, Jiaying Lu, Yao Ge, and Carl Yang. How can graph neural networks help document retrieval: A case study on cord19 with concept map generation. In European Conference on IR Research (ECIR) , 2022.

Hejie Cui, Jiaying Lu, Shiyu Wang, Ran Xu, Wenjing Ma, Shaojun Yu, Yue Yu, Xuan Kan, Tianfan Fu, Chen Ling, et al. A survey on knowledge graphs for healthcare: Resources, application progress, and promise. In ICML Workshop on Interpretable Machine Learning in Healthcare , 2023.

Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may not be faithful: Probing object hallucination in vision-language pre-training. In EACL , 2023.

Yang Ding, Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, and Qi Wu. Mukea: Multimodal knowledge extraction and accumulation for knowledge-based visual question answering. In CVPR , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

James Fan, Aditya Kalyanpur, David C Gondek, and David A Ferrucci. Automatic knowledge extraction from documents. IBM J Res Dev , 56:5-1, 2012.

Mikhail Galkin, Zhaocheng Zhu, Hongyu Ren, and Jian Tang. Inductive logical query answering in knowledge graphs. In NeurIPS , 2022.

Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In ACL-IJCNLP , 2021.

Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven CH Hoi. From images to textual prompts: Zero-shot vqa with frozen large language models. In CVPR , 2023.

Tao He, Lianli Gao, Jingkuan Song, and Yuan-Fang Li. Towards open-vocabulary scene graph generation with prompt-based finetuning. In ECCV , 2022.

Andreas Holzinger, Peter Kieseberg, Edgar Weippl, and A Min Tjoa. Current advances, trends and challenges of machine learning and knowledge extraction: from machine learning to explainable ai. In CD-MAKE , 2018.

Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput Surv , 55:1-38, 2023.

Justin Johnson, Andrej Karpathy, and Li Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In CVPR , 2016.

Xuan Kan, Hejie Cui, and Carl Yang. Zero-shot scene graph relation prediction through commonsense knowledge integration. In European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECML-PKDD) , 2021.

Dong-Jin Kim, Jinsoo Choi, Tae-Hyun Oh, and In So Kweon. Dense relational captioning: Triple-stream networks for relationship-based captioning. In CVPR , 2019.

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurIPS , 2022.

Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV , 123:32-73, 2017.

Hunter Lang, Monica N Agrawal, Yoon Kim, and David Sontag. Co-training improves promptbased learning for large language models. In ICML , 2022.

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping languageimage pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597 , 2023.

Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. BLIP: bootstrapping languageimage pre-training for unified vision-language understanding and generation. In ICML , 2022.

Manling Li, Alireza Zareian, Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng Ji, Shih-Fu Chang, Clare Voss, et al. Gaia: A fine-grained multimedia knowledge extraction system. In ACL , 2020.

Ye Liu, Hui Li, Alberto Garcia-Duran, Mathias Niepert, Daniel Onoro-Rubio, and David S Rosenblum. Mmkg: multi-modal knowledge graphs. In ESWC , 2019.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR , 2019.

Ximing Lu, Sean Welleck, Jack Hessel, Liwei Jiang, Lianhui Qin, Peter West, Prithviraj Ammanabrolu, and Yejin Choi. Quark: Controllable text generation with reinforced unlearning. NeurIPS , 2022.

Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In CVPR , 2022.

Sachit Menon and Carl Vondrick. Visual classification via description from large language models. In ICLR , 2023.

Jiaul H Paik. A novel tf-idf weighting scheme for effective ranking. In SIGIR , 2013.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. NeurIPS , 2019.

Sarah Pratt, Mark Yatskar, Luca Weihs, Ali Farhadi, and Aniruddha Kembhavi. Grounded situation recognition. In ECCV , 2020.

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS , 2015.

Fereshteh Sadeghi, Santosh K Kumar Divvala, and Ali Farhadi. Viske: Visual knowledge extraction and question answering by visual verification of relation phrases. In CVPR , 2015.

Maarten Sap, Ronan Le Bras, Emily Allaway, Chandra Bhagavatula, Nicholas Lourie, Hannah Rashkin, Brendan Roof, Noah A Smith, and Yejin Choi. Atomic: An atlas of machine commonsense for if-then reasoning. In AAAI , 2019.

Robyn Speer, Joshua Chin, and Catherine Havasi. Conceptnet 5.5: An open multilingual graph of general knowledge. In AAAI , 2017.

Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual-linguistic representations. In ICLR , 2020.

Yixuan Su, Tian Lan, Yan Wang, Dani Yogatama, Lingpeng Kong, and Nigel Collier. A contrastive framework for neural text generation. In NeurIPS , 2022.

Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. Unbiased scene graph generation from biased training. In CVPR , 2020.

Komal Teru, Etienne Denis, and Will Hamilton. Inductive relation prediction by subgraph reasoning. In ICML , 2020.

Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. NeurIPS , 2021.

Chenguang Wang, Xiao Liu, and Dawn Song. Language models are open knowledge graphs. arXiv preprint arXiv:2010.11967 , 2020.

Meng Wang, Sen Wang, Han Yang, Zheng Zhang, Xi Chen, and Guilin Qi. Is visual context really helpful for knowledge graph? a representation learning perspective. In ACM MM , 2021.

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. In NeurIPS , 2022.

Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. Neural text generation with unlikelihood training, 2019.

Danfei Xu, Yuke Zhu, Christopher Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR , 2017.

Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In CVPR , 2017.

Ran Xu, Yue Yu, Hejie Cui, Xuan Kan, Yanqiao Zhu, Joyce Ho, Chao Zhang, and Carl Yang. Neighborhood-regularized self-training for learning with few labels. In AAAI , 2023.

Ran Xu, Yue Yu, Joyce Ho, and Carl Yang. Weakly-supervised scientific document classification via retrieval-augmented multi-stage training. In SIGIR , pages 2501-2505, 2023.

Ran Xu, Yue Yu, Chao Zhang, Mohammed K Ali, Joyce C Ho, and Carl Yang. Counterfactual and factual reasoning over hypergraphs for interpretable clinical predictions on ehr. In Machine Learning for Health , 2022.

Yiben Yang, Chaitanya Malaviya, Jared Fernandez, Swabha Swayamdipta, Ronan Le Bras, Ji Ping Wang, Chandra Bhagavatula, Yejin Choi, and Doug Downey. Generative data augmentation for commonsense reasoning. In Findings of EMNLP , 2020.

Yue Yu, Kexin Huang, Chao Zhang, Lucas M Glass, Jimeng Sun, and Cao Xiao. Sumgnn: multityped drug interaction prediction via efficient knowledge graph summarization. Bioinformatics , 2021.

Yue Yu, Rongzhi Zhang, Ran Xu, Jieyu Zhang, Jiaming Shen, and Chao Zhang. Cold-start data selection for few-shot language model fine-tuning: A prompt-based uncertainty propagation approach. arXiv preprint arXiv:2209.06995 , 2022.

Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. arXiv preprint arXiv:2306.15895 , 2023.

Alireza Zareian, Svebor Karaman, and Shih-Fu Chang. Bridging knowledge graphs to generate scene graphs. In ECCV , 2020.

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. NeurIPS , 2022.

Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. In CVPR , 2019.

Rowan Zellers, Mark Yatskar, Sam Thomson, and Yejin Choi. Neural motifs: Scene graph parsing with global context. In CVPR , 2018.

Hongming Zhang, Xin Liu, Haojie Pan, Yangqiu Song, and Cane Wing-Ki Leung. Aser: A large-scale eventuality knowledge graph. In WWW , 2020.

Yiwu Zhong, Liwei Wang, Jianshu Chen, Dong Yu, and Yin Li. Comprehensive image captioning via scene graph decomposition. In ECCV , 2020.

Chunting Zhou, Junxian He, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Prompt consistency for zero-shot task generalization. In Findings of EMNLP , 2022.

Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR , 2022.

A Details of Data Augmentation with External Knowledge Resources

✔ Enhance Relation Recognition : We enriched the relationships between objects parsed from the original knowledge descriptions by leveraging the external resource of ConceptNet. ConceptNet comprises commonly observed entities and their connections, where edge weights signify the reliability and frequency of these relationships. The typical value of edge weights in ConceptNet is 1. To prevent the redundancy of common information and to maintain the validity of the enriched relations, we categorized the relationships based on their weights. Relationships with weights less than 1 were deemed 'weak' and those with a weight of 1 were labeled 'average'. We refrained from using these categories for relation enhancement. Instead, only relationships with weights greater than 1, indicative of high reliability, were employed for augmenting the relations.

✔ Boost Entity Perception : On the entity side, we augment complement entities and descriptive information with two external knowledge resources. On one hand, for descriptions with a high TFIDF+ score, we enrich related entities of the object from ConceptNet to create additional knowledge descriptions. The relatedness is based on the between-word relatedness score provided by ConceptNet and we take the threshold as 0.85. On the other hand, we employ the Commonsense Transformers (COMET) [4] model to enrich related new objects and descriptive information. The COMET model is a language model designed to generate commonsense knowledge and understand causal relationships between descriptions. It is pretrained using the atomic dataset, which consists of structured, crowdsourced knowledge about everyday events and their associated causes and effects. The COMET model can provide neighbor descriptions of the given input of nine different categories of relation. We take the xAttr and oEffect relation categories and augmented the COMET model by formulating the existing knowledge description texts as the input and choose the corresponding category branch during generation for enriching objects and descriptions respectively.

B Dataset Information

The statistic information of our augmented dataset is summarized in Table 8, where split specifies the dataset split, #image indicates the number of images in the split, #descriptor indicates the total number of relational descriptors of the images, #relation is the total number of unique relations in the relational descriptors after deduplication, and #subject & object is the total number of subjects and objects contained in the description text.

C Implementation Details

Table 10: Hyperparameters for training format-free visual knowledge generator.

Open relational region detector. The visual feature extraction backbone is constructed upon a pre-trained ResNet50-FPN. The detector head incorporates a BLIPbase equipped with the essential

ViT-B/16 for text supervision, using multiple fully connected layers to derive region features. For each candidate region, we engage a regressor to conduct boundary regression on these features. The detector undergoes fine-tuning for 20 epochs using the relational region bounding box dataset and an Adam optimizer [30]. The hyperparameters for training are detailed in Table 9.

Format-free visual knowledge generator. The format-free visual knowledge generator is initialized from BLIPbase, which incorporates the basic ViT-B/16. We fine-tune the generator model for 20 epochs using the same optimizer as the one employed for the region detector. Detailed hyperparameters for the visual knowledge generator can be found in Table 10.

D Human Evaluation Guidance and Interface

We perform the human evaluation on two of the four in-depth knowledge quality assessment metrics. We build an interface by referring to [50], where raters are presented with a given image and the corresponding knowledge descriptions and are required to choose one from the multiple choice for two questions on whether the knowledge is valid to humans and whether the knowledge description depicts the image. The detailed scoring criteria for Validity and Conformity are provided below:

Validity ( ↑ ): whether the generated visual knowledge is valid to humans .

-0 (Invalid): The knowledge description does not conform to human cognition, rendering it unreliable or misleading to humans.

-1 (Valid): The knowledge description is valid and accurately conforms to human cognition, providing reliable and meaningful knowledge to humans.

Conformity ( ↑ ): whether the generated knowledge faithfully depicts the scenarios in the images .

-0 (Inconsistent): The knowledge description does not faithfully depict the scenarios in the images, showing significant deviations or discrepancies, making it difficult for users to relate the textual information to the visual context.

-1 (Partially Conforming): The knowledge description partially conforms to the scenarios in the images, but there might be minor inconsistencies or missing relevant details.

-2 (Moderately Conforming): The knowledge description exhibits a moderate level of conformity with the scenarios in the images, capturing the key aspects and providing coherent descriptions.

-3 (Highly Conforming): The knowledge description highly conforms to the scenarios in the images, accurately capturing the details and faithfully representing the visual context.

Home

Rating Board

Examples

Rater1

Instructions for Test Users:

1. Assess the validity of each knowledge description by determining whether it

KnowledgeDescription(2/14):silverchairnext to counter

Ratethevalidityandconformityofeach

accurately represents human understanding.

knowledgedescription.

2. Assign a conformity score to each knowledge description, indicating the degree to

which it aligns with the image content.

·Is thisKnowledge Description valid?

3. Utilize the *previous’ or *next' buttons to navigate between knowledge descriptions.

The image will automatically change once all captions have been rated.

o (Not Valid)

Validity: whether the knowledge description is valid to human.

1 (Valid)

-0: Invalid  The knowledge description does not conform to human grammar, rendering

it unreliable or misleading to humans.

-1: Valid The knowledge description is valid and accurately conforms to human

· Is this Knowledge Description conform?

grammar, providing reliable and meaningful knowledge to humans.

Oo(Inconsistent Conforming)

Conformity: whether the knowledge description faithfully depicts the scenarios in the images.

-0: Inconsistent  The knowledge description does not faithfully depict the scenarios in

1(Partially Conforming)

the images, showing signifcant deviations or discrepancies, making it difhcult for users to

relate the textual information to the visual context.

O2(Moderately Conforming)

-1: Partially Conforming  The knowledge description partially conforms to the scenarios

in the images, but there might be minor inconsistencies or missing relevant details.

O3(Highly Conforming)

2: Moderately Conforming - The knowledge description exhibits a moderate level of

conformity with the scenarios in the images, capturing the key aspects and providing coherent

descriptions.

3: Highly Conforming  The knowledge description highly conforms to the scenarios in

Image_568.jpg

the images, accurately capturing the details and faithfully representing the visual context.

Agreement/validation We use Cohen's κ as the agreement score to measure potential subjectivity involved in ratings of knowledge quality. Cohen's κ is a statistic that is used to measure inter-rater reliability for qualitative items and is scaled from -1 (perfect systematic disagreement) to 1 (perfect agreement), where values ≤ 0 as indicating no agreement and 0.01-0.20 as none to slight , 0.21-0.40 as fair , 0.41-0.60 as moderate , 0.61-0.80 as substantial , and 0.81-1.00 as almost perfect agreement. Our calculated average pairwise Cohen's κ on human evaluation results from three different raters is 0.76, which indicates a good agreement.

E Parametric Knowledge Prompting Template

Given an image I and the corresponding extracted visual knowledge from it based on OpenVik , we perform knowledge comparison with parametric knowledge contained in LLM by prompting the gpt-3.5-turbo model with the object information contained in the I . The prompt format is shown in the followings:

Suppose you are looking at an image that contains the following subject and object entities:

Subject list:

Object

[Insert the subject names here]

list: [Insert the object names here]

Please extract 5-10 condensed descriptions that describe the interactions and/or relations among those entities in the image. Try to elucidate the associations and relationships with diverse language formats instead of being restricted to sub-verb-obj tuples.

F More Case Studies of Open Visual Knowledge from OpenVik

Figure 10 shows some other cases on the extracted open visual knowledge from OpenVik . In comparison to VG and Relational Caps, OpenVik exhibits superior performance at capturing novel entities , expanding object interactions through diverse relations , and enriching knowledge representation with nuanced descriptive details . For example for the bottom right image, OpenVik can extract novel entities such as ' tracks ', ' shoe ', diverse relations such as ' sticking out of ', and nuanced descriptive details such as ' cold thick ', ' with man feet on it ', ' brave '. The generated knowledge with a more format-free semantic structure is highlighted in red.

OpenVik :

green shrubbery growing along sidewalk purple parasol in rain holding by people

green trees in distance growing aside

green bush on side of building

people are trying not to get wet the woman look back in astonishment

OpenVik :

yellow tail of a red and yellow kite the people enjoying fun beach playing sands

flying a long tail kite has orange appearance

Visual Genome-Scene Graph:

<woman, has, hair> <person, wear, hat> <leaf, on, plant>

<man, in, suit>

Visual Genome-Region Descriptions: top of brown umbrella man in a suit man under an umbrella the people are enjoying their day

Relational Caps:

the people close to building the people are walking in a city the woman wearing blue jeans the woman has glasses

Visual Genome-Scene Graph:

<kite, in, sky>

<people, in, water>

<board, with, person>

<shirt, on, person>

Visual Genome-Region Descriptions:

the person wearing white shirt the woman on beach the person standing on beach the person in water

Relational Caps:

a blue board with a person someone built a sand castle the cloudy sky behind the kit white clouds in blue sky the woman running to flying kite excitedly

flying kite in the open blue sky the man looking at orange kite

OpenVik :

white and brown egg inside cardboard box the kale bunch wrapped using rope

green lettuce on crowded table white table cloth has flowers decorated

purple onion next to oranges very bright red polka dots on cloth

OpenVik :

white ground covered with cold thick snow

Visual Genome-Scene Graph:

<orange, on, table> <egg, in, carton> <pea, in, bag>

<onion, next to, limes>

Visual Genome-Region Descriptions:

a white colored egg seven oranges on a table green onion stalks that have been cut bunch of kale on table

Relational Caps:

the half dozen of white eggs green vegetables on table purple onion between orange the eggs in carton

Visual Genome-Scene Graph:

<tree, in, field>

<hat, on, head>

<grass, on, ground>

<shadow, of, skier>

Visual Genome-Region Descriptions:

snow divot in the hill writing on the pants lens on the sunglass ski attached to right foot

Relational Caps:

downhill skier wearing jacket the ski pole stuck in snow red hat worn on head the weeds in snow the shadow cast by brave skier

brown grass sticking out of snow-covered field the tracks in fresh snow with man feet on it

skiing man wearing yellow shoe young skier has red hat and smiling

G More Qualitative Examples on Applications

G.1 Text-to-Image Retrieval

Original text: Three young men playing Wii on a projection television. Three men laughing at some pictures from a projector. A group of gentleman playing video games in a dimly lit room. Some people chilling on the couch playing with a Nintendo Wii. A group of men playing a game with remote controllers.

Enriched text: men in group. men behind people. men playing. men in room playing video game. group of people. men in group are playing video game. people playing. people watching game. playing game.

Original text: An elderly woman sitting on the bench resting. An old woman leans on her back while sitting on an ornate bench. A woman is sitting on a bench near a fence. Older woman in dress sitting on a park bench. An old woman sitting on a bench next to a fence.

Enriched text: woman sitting on bench with a ornate. woman behind fence. woman wearing dress. woman in park. bench by fence. bench in park. woman in ornate dress on the bench. fence behind park.

Original text: A man is leaning over a fence offering food to an elephant. A man reaching out to an elephants trunk near a gate. A man is feeding an elephant over a fence. A man handing an elephant a stick in an enclosure at a zoo. A man reaches out to give the elephant something.

Enriched test: man behind fence. man next to trunk preparing food.  man holding stick in enclosure. man pointing at something. fence truck behind food. fence wrapped around trunk. fence behind elephant. fence made of stick. fence surrounds enclosure. trunk of elephant. elephant in enclosure.

Original text: A row of parked motorcycles sitting in front of a tall building. A stone street with bicycles and motor bikes parked on the side and people standing on the sidewalks in front of buildings. Cityscape of pedestrians enjoying an old European city. a row of bikes and mopeds is parked along the street. Motorcycles and mopeds line a side street during the day in a city.

Enriched text: row made of stone leading into city. motor in row. row of people. street made of stone. wall made of stone next to side. stone wall behind people. people in line crossing street. street in city. motor on side. people riding motor in city. motor in line. people in line in city. day at city.

Original text: A herd of cattle is feeding at the river's edge. Many cows next to a body of water in a field. A herd of cows grazes in a field near a river. A herd of cattle standing in grassy area next to water. A herd of cattle is near a flock of birds swimming in the water.

Enriched text: herd of cattle crossing river. herd traveling by water. cattle crossing river. cattle in field. river across field in front of area. water near field. water near area. water next to flock. Birds inside of water. flock in field.

Original text: A white refrigerator freezer sitting inside of a kitchen. A corner of a kitchen with a big fridge. A kitchen has a plain white fridge in the corner. A refrigerator in the corner of a kitchen just off the dining room a room showing a very big fridge and a dining table.

Enriched text: refrigerator has freezer. refrigerator in corner. refrigerator in bright kitchen. refrigerator in room. refrigerator next to table sitting in kitchen. freezer next to table. corner window in room. corner of table. fridge in kitchen. table in kitchen. fridge table next to table in room.

Figure 11 presents more qualitative examples of OpenVik -based visual knowledge enrichment on captions. The enriched text is based on the objects present in the images themselves, supplemented with additional relationships from our generated visual knowledge in OpenVik . It is shown that the introduced relationships often provide new context information that aligns with the visual content of the images. For example, in the image of an old woman sitting on a bench in a park, the enriched context information includes the positional relationship between the ' bench ', ' fence ', and ' park ', which provides a more comprehensive description of the original image.

G.2 Grounded Situation Recognition

Figure 12 presents more qualitative examples of OpenVik -based context enrichment in the grounded situation recognition (GSR) task. Our context enrichment setting for the GSR task is to perform enrichment based on verbs like ' shopping ' and ' carrying '. We further restrict the enriched context with the objects contained in the image to avoid noisy enrichment. For example, for the image showing people shopping at a market, the enriched knowledge contexts could be ' the people shopping at market ', ' standing person shopping for fruit '. The idea is to enrich the original description T : ' An image of <verb> ' with relevant actions and relations with the extracted visual knowledge from OpenVik , which can potentially help in drawing-in the matched candidates.

G.3 Visual Commonsense Reasoning

Figure 13 presents more qualitative examples of OpenVik -based context enrichment in the visual commonsense reasoning (VCR) task. The context enrichment on VCR is performed at two-level,

Verb:

shopping

Origin text:

This is an image

of shopping

Enrich text:

the people

shopping at market. standing

person shopping for fruit.

Verb:

spraying

Original text: This is an image of spraying. Enriched text: the water spraying from fountain. the water spraying from spout. the water spraying in park.

Verb:

walking

Original text:

This is an image

of walking.

Enriched text: the person walking through forest. the people walking on sidewalk.

Verb:

typing

Original text:

This is an image

of typing.

Enriched text:

sitting woman

typing on smart

open laptop.

Verb:

carving

Original text:

This is

an image of carving.

Enriched text:

wood

carving in center. man

carving wood.

Verb:

carrying

Original text:

This is an image

of carrying.

Enriched text: walking person carrying bag. man carrying hay in the field.

Verb:

licking

Original text:

This is an image of licking.

Enriched text:

black dog licking food.

incorporating both entities and relations: (1) we parse the question and options to obtain all (S, O) pairs and, for each entity pair, apply the same relation augmentation as in the image retrieval task; (2) for the V in each option, we enrich the visual context using the same method as illustrated in GSR. It is shown that unrelated answers are usually enriched with contexts that are not relevant to the image, thus enlarging the distance between incorrect answers and the question, e.g., the enriched contexts ' squating person fixing handy bathroom ' for example 3 in Figure 13. At the same time, the knowledge description of the correct answer is enhanced by incorporating information that aligns with the image contents, e.g., the enriched knowledge contexts ' sitting people on red ground ' for example 1 in Figure 13.

H Full List of Filtered Verbs for GSR

We provide the full list of verbs out of the predefined 504 candidates of GSR [36] that can be accurate-matched or fuzzy-matched to extracted visual knowledge in Table 11, based on which we compose the testing subset for our evaluation on GSR application in Section 5.2.

personl]

[bottlel]

person2

person3

Question: Where is' Person1 sitting?

A

He is in a laboratory.

B

C

He is sitting at a bar. the person sitting behind sneaky barrier.

In a fort in his house. the person walking by light house.

D

He is sitting on the ground. sitting person on red ground.

Answer: D He is sitting on the ground.

[person2]

[personl]

[apple1]

[orange1]

[cake1]

[bowl1]

Question: Where is Person2 going?

A

Person2 is going into the store. the person walking into store.

B

Person2 is getting into a carriage. sitting person inside carriage.

C Person1 is going to the bathroom. squating person fixing handy bathroom.

D

Person1 is going outside to play after the conversation with Person2 is over.

Answer: A Person2 is going into the store.

person10]

perso

[per[person18]

[person12]rson6]

son5

gerson8]

person7]

[personl]

personlii

[person16]

[handbag1]

persong

pe

SOT

person17]

person15]

Question: Why is  Person7 in motion?

A

Person14 is running desperately.

B

C

D

Person7 is climbing over the boat. the person standing inside white boat.

Person7 is walking fast to the bathroom. squating person fixing handy bathroom.

Person7 is going to try to protect  Person10 from a threat. Person7 is moving forward to challenge what ever could be there.

Answer: B Person7 is climbing over the boat.

person2

[persont]

[tie1]

Question : Where are Person1 and Person2?

A Person1 and Person2 are sitting outside of a general store. the person walking by store.

B Person1 and Person2 are standing on top of a train car. jumping person on top board. walking person next to white train. the person walking near active car. yellow train sitting atop track. sliced carrot on top counter red car of old train.

C

Person1 and Person2 are in an office. walking person outside office.

D

Person1 and Person2 are in the kitchen. the person eating in hungry kitchen.

Answer: C Person1 and Person2 are in an office.

[person2]

[personl]

diningtablel]

chair1]

[diningtable2]

Question :  What will  Person2 do next?

A

Person2 will speak angrily at  diningtable2, then walk off.

B

C

D

Person2 will sit down on  chair1. painting person near giant chair.

Person2 will feed  bowl1. the person skate boarding in a athletic bowl.

Person2 will open the box. the person holding a box full of oranges.

Answer: B Person2 will sit down on  chair1.

[persons]

person2

[person3]

personl

[person4]

[person6]

person7

Question: What is  Person1 doing here?

A He is in prison serving a prison sentence. person writing sentences.

B

He is trying to get information. person gaining information.

C

Person1 is a waiter. person talking with waiter in restaurant.

D He is existing a building. walking person near large building. Answer: C Person1 is a waiter.

Matching Type

Accurate

Fuzzy

The Word List of Event Types

putting, butting, bathing, dusting, rearing, turning, skating, placing, carting, staring, biting, mashing, folding, wetting, sprinkling, branching, drying, standing, flaming, taxiing, performing, circling, molding, parachuting, glowing, fishing, drinking, speaking, pawing, blocking, milking, racing, stripping, potting, spinning, eating, making, kicking, catching, lacing, urinating, sleeping, pressing, buttering, shearing, sliding, hiking, glaring, dipping, swimming, shopping, slicing, shelling, wagging, grilling, crafting, raining, clawing, splashing, rubbing, snowing, breaking, guarding, clipping, sewing, braiding, telephoning, buttoning, waiting, serving, picking, camping, leaning, working, kissing, wrapping, trimming, tripping, pasting, soaring, driving, kneeling, pumping, coloring, lighting, training, ducking, bowing, arching, cooking, checking, pushing, flipping, rocking, cresting, cleaning, reading, nailing, stitching, building, climbing, covering, shelving, attaching, calming, selling, gluing, dyeing, lapping, photographing, peeling, sprouting, licking, displaying, combing, stacking, planting, fastening, buying, mopping, burning, erasing, measuring, dining, tattooing, gardening, decorating, clearing, fixing, weeding, pulling, feeding, watering, crowning, shaking, dripping, emptying, typing, chasing, poking, leaping, pouring, hanging, sniffing, piloting, falling, overflowing, resting, crashing, carving, ballooning, wading, loading, shaving, boarding, pinning, rowing, juggling, shoveling, hugging, throwing, calling, singing, carrying, walking, writing, crouching, floating, painting, opening, tying, riding, strapping, dialing, saying, bubbling, signing, camouflaging, operating, leading, laughing, parading, skiing, drawing, gnawing, celebrating, spreading, filling, giving, running, smelling, plowing, helping, brushing, scooping, adjusting, wrinkling, steering, biking, smiling, spraying, boating, paying, chewing, stuffing, clinging, landing, wheeling, talking, scoring, teaching, jogging, pitching, flapping, tipping, scrubbing, sitting, surfing, stirring, competing, drumming, jumping, filming, dancing, waxing, hitting, recording, baking, waving, washing, signaling, chopping, stretching, rafting, microwaving, phoning, lifting, swinging, releasing, ramming, towing, packing, hauling, frying ( 244 words )

educating, marching, spanking, descending, smearing, heaving, cramming, inflating, stooping, inserting, squeezing, tugging, tilting, moistening, swarming, subduing, waddling, winking, flexing, punching, attacking, nuzzling, sprinting, sucking, puckering, sketching, rotting, videotaping, complaining, tuning, locking, hurling, pricking, arranging, constructing, slapping, sweeping, restraining, dousing, frisking, twisting, wringing, hoisting, immersing, shredding, blossoming, igniting, spying, offering, pouting, confronting, docking, assembling, prying, grinning, sharpening, pruning, disciplining, nipping, coaching, nagging, storming, handcuffing, apprehending, bouncing, clenching, taping, distributing, striking, studying, plunging, curling, aiming, sowing, grinding, rinsing, punting, mowing, hitchhiking, skipping, leaking, providing, hunching, spoiling, kneading, burying, foraging, lathering, vaulting, ejecting, mending, pinching, deflecting, ascending, peeing, bothering, repairing, pedaling, ailing, fueling, skidding, scraping, soaking, grimacing, scolding, spitting, knocking, crushing, bandaging, saluting, fording, stumbling, discussing, raking, launching, whirling, fetching, brawling, retrieving, snuggling, exercising, colliding, stroking, whipping, tilling, betting, farming, browsing, examining, dropping, barbecuing, ignoring, asking, flinging, perspiring, embracing, slipping, flicking, smashing, arresting, lecturing, tearing, gasping, applying, counting, spilling, dragging, recovering, practicing, scratching, shooting, packaging, hunting, stinging ( 154 words )

Peer reviews:
comments: SUMMARY:
The authors claim that they propose OpenVik which consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest. However, some issues are concerned.

STRENGTHS:
The authors claim that they propose OpenVik which consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest.

WEAKNESSES:
1. Paper has weak technical depth, it requires more technical details. 

2. Further experimental analysis needed. Please give qualitative validation.

QUESTIONS:
1. Paper has weak technical depth, it requires more technical details. 

2. Further experimental analysis needed. Please give qualitative validation.

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors present OpenVik, a method for open visual knowledge extraction. It consists of an open relational region detector to detect regions potentially containing relational knowledge and a visual knowledge generator that generates format-free knowledge by prompting the large multimodality model with the detected region of interest. They explore two data enhancement techniques for diversifying the generated format-free visual knowledge. Extensive knowledge quality evaluations highlight the correctness and uniqueness of the extracted open visual knowledge by OpenVik. Moreover, they show consistent improvements across various visual reasoning applications, indicating the real-world applicability of OpenVik.\n\nSTRENGTHS:\n- Well-written\n- Extensive Evaluations (Ablation Study and Applications)\n- OpenVik outperforms traditional approaches and generates rich Scene Graphs\n\nWEAKNESSES:\n- The importance of the pre-training for the Open Relational Region Detector is mentioned but not evaluated\n\n- It seems that the performance of OpenVik is mainly based on BLIP's performance\nAblation of OpenVik vs. BLIP is missing.\n\n- Usefulness of Scene Graph needs to be evaluated instead of diversity\n\nQUESTIONS:\n- How useful are the generated facts?\nThere seems to be a large variance in the entities, relations and the overall details in the facts. When looking at the produced facts, it can be seen that the facts generated for individual image patches could be quite useless and random, e.g. \"dark brown mane growing behind head\", and also not related to the main entities in the image, e.g. zebra.\nThere is no real structure in the facts. \n\n- Do the generated facts suffer from typical problems like hallucination?\nIf this is the case authors should also elaborate on that.\n\nLIMITATIONS:\nThe paper should elaborate more on the difficulties and the limitations of their approach.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to visual knowledge extraction named OpenVik. It comprises three main components: the Open Relational Region Detector, the Format-Free Visual Knowledge Generator, and the Diversity-Driven Data Enhancement module.\n\nThe Open Relational Region Detector, built on the object detection framework FasterRCNN, is fine-tuned to discern relation-oriented visual regions in images. The Format-Free Visual Knowledge Generator's purpose is to utilize this relational region information to create more relation-relevant visual knowledge from an image. The last component, the Diversity-Driven Data Enhancement module, functions as a knowledge post-processing module. Its primary goals include filtering out extraneous knowledge facts and enriching relations via synonym substitution, leveraging external knowledge resources.\n\nThe experimental results indicate that OpenVik surpasses previous benchmarks in most test scenarios, implying its effectiveness in generating high-quality relational knowledge from images.\n\nSTRENGTHS:\n1. The visual representation of the data through figures and charts is commendable. In particular, Figure 1 effectively illustrates the primary components of OpenVik. The color-coded highlights for 'entities', 'relations', and 'descriptive details' significantly enhance the readability of the examples.\n\n2. The architectural design of the model, along with the selected loss functions, is robust and well-conceived. The three components harmoniously co-function to train on current relational datasets and subsequently generate novel open visual knowledge from images.\n\n3. The level of detail in the explanation of the proposed method is noteworthy. The comprehensive presentation allows readers to gain a thorough understanding of OpenVik even after a single read-through.\n\nWEAKNESSES:\n1. The paper could benefit from a stronger justification for its approach. With the existence of established multimodal models like BLIP[1], mPLUG[2], OFA[3], and recent advancements like BLIP-2[4], MiniGPT4[5], all of which can potentially perform Visual Knowledge Extraction through prompting, the specific need for OpenVik is not explicitly clarified. Further explanation about what unique advantages OpenVik brings would be beneficial.\n\n2. The comparative analysis in section 4.1 appears unbalanced. The use of older models, such as the captioning model and relational captioning model, for benchmarking does not provide a robust basis for comparison. It would be more persuasive if contemporary models were included in the comparison.\n\n3. The ablation study appears to be not comprehensive enough. Crucial distinctions, such as those between the Open Relational Region Detector and the standard FasterRCNN, are not sufficiently elucidated. More detailed examinations of such comparisons would significantly enhance the paper's depth of analysis.\n\n4. The paper lacks clear explanation regarding the evaluation settings, such as the specific split used during evaluation. While I did find these details eventually in the supplementary material, I would recommend the authors incorporate such essential information directly within the main body of the paper. This would ensure a smoother reading experience and easier access to important technical specifications.\n\n[1] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation\n\n[2] mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections\n\n[3] OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework\n\n[4] BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models\n\n[5] MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models\n\nQUESTIONS:\n1. With the existence of general multimodal models, why do we need OpenVik?\n\nPlease answer my questions mentioned in the **Weaknesses** .\n\nLIMITATIONS:\nGood", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper that introduces a new paradigm of open visual knowledge extraction called OpenVik. This method generates format-free knowledge by prompting a large multimodality model with detected regions of interest. The proposed framework consists of an open relational region detector and a format-free visual knowledge generator. The paper highlights the limitations of existing approaches to visual knowledge extraction and demonstrates the correctness and uniqueness of the extracted open visual knowledge by OpenVik. The extracted knowledge is integrated across various visual reasoning applications, showing consistent improvements and indicating the real-world applicability of OpenVik.\n\nSTRENGTHS:\n1. Novel Approach: The paper introduces a new paradigm, OpenVik, for open visual knowledge extraction. This system generates format-free knowledge by prompting a large multimodality model with detected regions of interest, which is a novel approach in the field.\n2. Comprehensive Evaluation: The paper provides a thorough evaluation of the generated knowledge, using traditional generative metrics and in-depth knowledge quality assessment. This comprehensive evaluation helps validate the effectiveness of the proposed method.\n3. Comparison with Existing Knowledge Sources: The paper compares the extracted visual knowledge with non-parametric knowledge in existing knowledge graphs and parametric knowledge from large language models. This comparison provides a clear understanding of the unique value proposition of OpenVik.\n4. Real-World Application: The extracted knowledge is integrated across various visual reasoning applications, showing consistent improvements. This indicates the real-world applicability of OpenVik, making it a practical solution for visual knowledge extraction.\n\nWEAKNESSES:\n1. Limited Dataset: The training data for the model is built based on Visual Genome and its relation-enhanced version Dense Relational Captioning. The performance of the model on more diverse datasets remains to be studied, which could limit its generalizability.\n2. Implementation Complexity: The model involves complex implementation details, including the use of a ResNet50-FPN backbone for the open relational region detector and fine-tuning of the visual knowledge extractor. This complexity could make the model difficult to implement and adapt for other researchers or practitioners.\n3. Limited comparisons and lack of ablation studies: The authors only compare to a few baselines, but there are many knowledge-enhanced models which need to be compared in the application section.  \n\nOther detailed questions are listed in Questions section.\n\nQUESTIONS:\n1. There are many other notions in Figure 1, such as L_k, can the authors explain all of them or list all the equations?\n2. What is binary mask (line 113) in the format-free visual knowledge generator?\n3. What is the training loss of the method? It is not clear to me, since there are many notions in Figure 1 but not appearing in the paper.\n4. Can this method be scale to larger training dataset, or other backbones?\n5. In the evaluation of in-depth knowledge quality, the authors mentioned they randomly selected 100 images. How about the compared method, did they use the same images or same number of images? How many raters or annotators in the studies?\n6. In the application section, are the results of OpenVik zero-shot or fine-tuning? The implement details are not clear in the applications, and need to compare with more other methods.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new paradigm called open visual knowledge extraction and designs a framework OpenVik to generate format-free knowledge instead of pre-defined format knowledge. The authors also present two data enhancement technologies to ensure the diversity of knowledge. Moreover, the paradigm could also be integrated with other applications to jointly boost the performance.\n\nSTRENGTHS:\n1. The paper is well-written and easy to understand. \n2. The proposed open relational region detector is interesting.\n3. The evaluation for in-depth knowledge quality is significant.\n\nWEAKNESSES:\n1. The originality of the work is incremental. Indeed, both open relational region detector and format-free visual knowledge generator are minor modification existing models. \n2. The experiments presented in this paper are insufficient. The comparison baselines should be more explored because the task of this paper is quite similar to region captioning task, such as [1], [2].\n3. The word ‘open’ is confusing in the title. The paper could not tackle the open-world knowledge. It is still based on the close-set of objects.\n4. The implementation of open relational region detector lacks the detailed information, such as the update of bounding box. The label set raises another question. \n\nReferences\n[1] Zhong, Yiwu, et al. \"Comprehensive image captioning via scene graph decomposition.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XIV 16. Springer International Publishing, 2020.\n[2] Ghosh, Shalini, et al. \"Generating natural language explanations for visual question answering using scene graphs and visual attention.\" arXiv preprint arXiv:1902.05715 (2019).\n\nQUESTIONS:\n1. If the supervision is the generated description, how could it guarantee the correspond label for the region that contains the same objects and relations? Is it a few-shot task?\n2. More detail of generator should be provided. What is the input of generator? Is the single region or the whole image?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}