SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Open relational region detector. The visual feature extraction backbone is constructed upon a pre-trained ResNet50-FPN. The detector head incorporates a  $\mathrm{BLIP}_{\mathrm{base}}$  equipped with the essential
ViT-B/16 for text supervision, using multiple fully connected layers to derive region features. For each candidate region, we engage a regressor to conduct boundary regression on these features. The detector undergoes fine-tuning for 20 epochs using the relational region bounding box dataset and an Adam optimizer [30]. The hyperparameters for training are detailed in Table 9.
Format-free visual knowledge generator. The format-free visual knowledge generator is initialized from  $\mathrm{BLIP}_{\mathrm{base}}$ , which incorporates the basic ViT-B/16. We fine-tune the generator model for 20 epochs using the same optimizer as the one employed for the region detector. Detailed hyperparameters for the visual knowledge generator can be found in Table 10.
D Human Evaluation Guidance and Interface
We perform the human evaluation on two of the four in-depth knowledge quality assessment metrics. We build an interface by referring to [50], where raters are presented with a given image and the corresponding knowledge descriptions and are required to choose one from the multiple choice for two questions on whether the knowledge is valid to humans and whether the knowledge description depicts the image. The detailed scoring criteria for Validity and Conformity are provided below:
- Validity  $(\uparrow)$ : whether the generated visual knowledge is valid to humans.
- 0 (Invalid): The knowledge description does not conform to human cognition, rendering it unreliable or misleading to humans.
- 1 (Valid): The knowledge description is valid and accurately conforms to human cognition, providing reliable and meaningful knowledge to humans.
- Conformity ( $\uparrow$ ): whether the generated knowledge faithfully depicts the scenarios in the images.
- 0 (Inconsistent): The knowledge description does not faithfully depict the scenarios in the images, showing significant deviations or discrepancies, making it difficult for users to relate the textual information to the visual context.
- 1 (Partially Conforming): The knowledge description partially conforms to the scenarios in the images, but there might be minor inconsistencies or missing relevant details.
- 2 (Moderately Conforming): The knowledge description exhibits a moderate level of conformity with the scenarios in the images, capturing the key aspects and providing coherent descriptions.
- 3 (Highly Conforming): The knowledge description highly conforms to the scenarios in the images, accurately capturing the details and faithfully representing the visual context.
Figure 9: The human evaluation interface for in-depth knowledge quality evaluation.
Agreement/Validation We use Cohen's  $\kappa$  as the agreement score to measure potential subjectivity involved in ratings of knowledge quality. Cohen's  $\kappa$  is a statistic that is used to measure inter-rater reliability for qualitative items and is scaled from -1 (perfect systematic disagreement) to 1 (perfect agreement), where values  $\leq 0$  as indicating no agreement and 0.01-0.20 as none to slight, 0.21-0.40 as fair, 0.41-0.60 as moderate, 0.61-0.80 as substantial, and 0.81-1.00 as almost perfect agreement. Our calculated average pairwise Cohen's  $\kappa$  on human evaluation results from three different raters is 0.76, which indicates a good agreement.
E Parametric Knowledge Prompting Template
Given an image  $\mathcal{I}$  and the corresponding extracted visual knowledge from it based on OpenVik, we perform knowledge comparison with parametric knowledge contained in LLM by prompting the gpt-3.5-turbo model with the object information contained in the  $\mathcal{I}$ . The prompt format is shown in the followings:
Suppose you are looking at an image that contains the following subject and object entities:
Subject list: [Insert the subject names here]
Object list: [Insert the object names here]
Please extract 5-10 condensed descriptions that describe the interactions and/or relations among those entities in the image. Try to elucidate the associations and relationships with diverse language formats instead of being restricted to sub-verb-object tuples.
F More Case Studies of Open Visual Knowledge from OpenVik
Figure 10 shows some other cases on the extracted open visual knowledge from OpenVik. In comparison to VG and Relational Caps, OpenVik exhibits superior performance at capturing novel entities, expanding object interactions through diverse relations, and enriching knowledge representation with nuanced descriptive details. For example for the bottom right image, OpenVik can extract novel entities such as "tracks", "shoe", diverse relations such as "sticking out of", and nuanced descriptive details such as "cold thick", "with man feet on it", "brave". The generated knowledge with a more format-free semantic structure is highlighted in red.
□ green shrubbery growing along sidewalk
purple parasol in rain holding by people
green trees in distance growing aside
□ green bush on side of building
people are trying not to get wet
□ the woman look back in astonishment
<woman, has, hair>
<person, wear, hat>
<leaf.on.plant>
<man,in,suit>
Visual Genome-Region Descriptions
top of brown umbrella
man in a suit
man under an umbrella
the people are enjoying their day
the people close to building
the people are walking in a city
the woman wearing blue jeans
the woman has glasses
white and brown egg inside cardboard box
the kale bunch wrapped using rope
green lettuce on crowded table
white table cloth has flowers dec
purple onion next to oranges very
red polka dots on cloth
<orange, on, table>
<egg, in, carton>
<pea, in, bag>
<onion, next to, lines>
a white colored egg
seven oranges on a table
green onion stalks that have been cut
bunch of kale on table
OpenVik:
Figure 10: Case studies of open visual knowledge from OpenVik.
yellow tail of a red and yellow kite
the people enjoying fun beach playing sands
□ flying a long tail kite has orange appearance
<kite, in, sky>
<people, in, water>
<board, with, person>
<shirt, on, person>
Visual Genome-Region Descriptions:
the person wearing white shirt
the woman on beach
the person standing on beach
the person in water
a blue board with a person
someone built a sand castle
the cloudy sky behind the kit
white clouds in blue sky
OpenVik
white ground covered with cold thick snow
brown grass sticking out of snow-covered f
the tracks in fresh snow with man feet on it
the half dozen of white eggs
green vegetables on table
purple onion between orange
the eggs in carton
<tree, in, field>
<hat, on, head>
<grass, on, ground>
<shadow, of, skier>
Visual Genome-Region Descriptions:
snowdivot in the hill
writing on the pants
lens on the sunglass
ski attached to right foot
downhill skier wearing jacket
the ski pole stuck in snow
red hat worn on head
the weeds in snow
the shadow cast by brave skier
skiing man wearing yellow shoe
young skier has red hat and smiling
G More Qualitative Examples on Applications
G.1 Text-to-Image Retrieval
Figure 11: Qualitative examples of OpenVik context enrichment on text-to-image retrieval.
Figure 11 presents more qualitative examples of OpenVik-based visual knowledge enrichment on captions. The enriched text is based on the objects present in the images themselves, supplemented with additional relationships from our generated visual knowledge in OpenVik. It is shown that the introduced relationships often provide new context information that aligns with the visual content of the images. For example, in the image of an old woman sitting on a bench in a park, the enriched context information includes the positional relationship between the "bench", "fence", and "park", which provides a more comprehensive description of the original image.
G.2 Grounded Situation Recognition
Figure 12 presents more qualitative examples of OpenVik-based context enrichment in the grounded situation recognition (GSR) task. Our context enrichment setting for the GSR task is to perform enrichment based on verbs like "shopping" and "carrying". We further restrict the enriched context with the objects contained in the image to avoid noisy enrichment. For example, for the image showing people shopping at a market, the enriched knowledge contexts could be "the people shopping at market", "standing person shopping for fruit". The idea is to enrich the original description  $\mathcal{T}$ : "An image of <verb>" with relevant actions and relations with the extracted visual knowledge from OpenVik, which can potentially help in drawing-in the matched candidates.
G.3 Visual Commonsense Reasoning
Figure 13 presents more qualitative examples of OpenVik-based context enrichment in the visual commonsense reasoning (VCR) task. The context enrichment on VCR is performed at two-level,
Verb: shopping
Origin text: This is an image of shopping.
Enrich text: the people shopping at market. standing person shopping for fruit.
Verb: licking
Original text: This is an image of licking. Enriched text: black dog licking food.
Verb: carving
Original text: This is an image of carving. Enriched text: wood carving in center. man carving wood.
Verb: spraying
Original text: This is an image of spraying. Enriched text: the water spraying from fountain, the water spraying from spout, the water spraying in park.
Figure 12: Qualitative examples of OpenVik context enrichment on task GSR.
Verb: walking  
Original text:  
This is an image of walking.  
Enriched text: the person walking through forest. the peop walking on sidewalk.
Verb: typing  
Original text:  
This is an image of typing.  
Enriched text:  
sitting woman typing on smart open laptop.
Verb: carrying  
Original text:  
This is an image of carrying.  
Enriched text:  
walking person  
carrying bag, man  
carrying hay in the field.
incorporating both entities and relations: (1) we parse the question and options to obtain all  $(S, 0)$  pairs and, for each entity pair, apply the same relation augmentation as in the image retrieval task; (2) for the V in each option, we enrich the visual context using the same method as illustrated in GSR. It is shown that unrelated answers are usually enriched with contexts that are not relevant to the image, thus enlarging the distance between incorrect answers and the question, e.g., the enriched contexts "squatting person fixing handy bathroom" for example 3 in Figure 13. At the same time, the knowledge description of the correct answer is enhanced by incorporating information that aligns with the image contents, e.g., the enriched knowledge contexts "sitting people on red ground" for example 1 in Figure 13.
H Full List of Filtered Verbs for GSR
We provide the full list of verbs out of the predefined 504 candidates of GSR [36] that can be accurate-matched or fuzzy-matched to extracted visual knowledge in Table 11, based on which we compose the testing subset for our evaluation on GSR application in Section 5.2.
Question: Where is' Person1 sitting?
A He is in a laboratory.
B He is sitting at a bar. the person sitting behind sneaky barrier.
C In a fort in his house, the person walking by light hou
D He is sitting on the ground. Sitting person on red ground.
Answer: D He is sitting on the ground.
Question: Where is Person2 going?
A Person2 is going into the store. the person walking into store.
B Person2 is getting into a carriage. sitting person inside carriage.
C Person1 is going to the bathroom. squating person fixing handy bathroom.
D Person1 is going outside to play after the conversation with Person2 is over.
Answer: A Person2 is going into the store.
Question: Why is Person7 in motion?
A Person14 is running desperately.
B Person7 is climbing over the boat. the person standing inside white boat
C Person7 is walking fast to the bathroom. squatting person fixing handy bathroom.
D Person7 is going to try to protect Person10 from a threat. Person7 is moving
forward to challenge what ever could be there
Answer: B Person7 is climbing over the boat.
Question: What will Person2 do next?
A Person2 will speak angrily at diningtable2, then walk off.
B Person2 will sit down on chair1. painting person near giant chair.
C Person2 will feed bowl1. the person skate boarding in a athletic bowl.
D Person2 will open the box. the person holding a box full of oranges.
Answer: B Person2 will sit down on chair1.
Question: Where are Person1 and Person2?
Figure 13: Qualitative examples of OpenVik context enrichment on task VCR.
A Person1 and Person2 are sitting outside of a general store. the person walking by store.
B Person1 and Person2 are standing on top of a train car. jumping person on top board, walking person next to white train. the person walking near active car. yellow train sitting atop track, sliced carrot on top counter red car of old train.
C Person1 and Person2 are in an office. walking person outside office.
D Person1 and Person2 are in the kitchen. the person eating in hungry kitchen. Answer: C Person1 and Person2 are in an office.
Question: What is Person1 doing here?
A He is in prison serving a prison sentence. person writing sentences.
B He is trying to get information. person gaining information
C Person1 is a waiter. person talking with waiter in restaurant.
D He is existing a building. walking person near large building.
Answer: C Person1 is a waiter.