Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?
Authors: Yang Chen
Hexiang Hu
Yi Luan
Haitian Sun
Soravit Changpinyo
Alan Ritter
Ming-Wei Chang
Venue: EMNLP 2023 Main (2023)
Keywords: visual question answering
vision and language
pre-trained multimodal model

Abstract: Pre-trained vision and language
models have demonstrated state-of-the-art capabilities over existing tasks involving images and texts, including visual question answering. However, it remains unclear whether these models possess the capability to answer questions that are not only querying visual content but knowledge-intensive and information-seeking. In this study, we introduce InfoSeek, a visual question answering dataset tailored for information-seeking questions that cannot be answered with only common sense knowledge. 
Using InfoSeek, we analyze various pre-trained visual question answering models and gain insights into their characteristics. Our findings reveal that state-of-the-art pre-trained multi-modal models (e.g., PaLI-X, BLIP2, InstructBLIP) face challenges in answering visual information-seeking questions, but fine-tuning on the InfoSeek dataset elicits models to use fine-grained knowledge that was learned during pre-training.
Furthermore, we show that accurate visual entity recognition can be used to improve performance on InfoSeek by retrieving relevant documents, showing a significant space for improvement.

Can Pre-trained Vision and Language Models Answer
Visual Information-Sseking Questions?
Yang Chen V Hexiang Hu
Yi Luan
Haitian Sun
Soravit Changpinyo
Alan Ritter
Ming-Wer Wei Chang
Google Deepmind
Google Research
Georgia Institute of Technology
Abstract
Q: What days might most commonly go to this building?
Pre-trained vision and language models (Chen al., 2023b,a; Dai al., 2023; et al., 2023b) have demonstrated state-of-the-art pabilities over existing tasks involving images and texts, including visual question answer- ing. However, it remains unclear whether these models possess the capability to answer ques- only querying visual con- intensive and information-
A: Sunday
Previous VQA
Q: Who designed this building? A: Antonio Barluzzi
Q: which year was this building constructed?
A: 1955
INFOSEEK
tions that tent but seeking. this study, we introduce INFOS- EEK!, visual question answering dataset tai- lored for information-seeking questions that cannot answered with only common sense knowledge. Using INFOSEEK, various pre-trained visual question answering models and gain insights into their character- istics. Our findings reveal that state-of-the-ant pre-trained multi-modal models (e.g.. PaLI-X. BLIP2, etc.) face challenges in answering vi- sual informatioonsseeiing questions, but fine- tuning on the INFOSEEK dataset elicits models fine- gnainedieed knowledge that learned during their pre-training. Furthermore, we show accurate visual entity recognition can be used to improve performance on INFOSEEK by retrieving relevant documents, showing a significant space for improvement.
While 70.8% OK- VQA questions can be answered by average adults without using search en- gine, INFOSEEK poses challenges to query fine- grained information about the visual entity (e.g.. Dominus Flevit Church), resulting in sharp drop to 4.4% ($2).
Figure 1:
analyze
after taking picture of the specific church in Fig- ure person might want to know the date of construction, who decorated the interior of the church. Although the entity is presented in the im- age (the specific church), the relevant knowledge (e.g., the date) is not. Given recent advances on pre-trained visual and language models (Alayrac et al., 2022; Chen et al., 2023b; Li et al., 2023b), do these models also understand how to answer visual information-seekingg questions?
To study this research question, a visual question answering (VQA) dataset focusing on info-seeking questions is inevitably required. However, not all VQA datasets meet this criterion. For example. by design, the majority of questions in datasets such as VQA v2 (Goyal et al., 2017) focus on vi- sual attributes and object detection that does not require information beyond the image to answer. While models capable of answering these types of questions have the potential to aid visually im- paired individuals (Gurari et al., 2018), there a broader class of info-seeking questions that can- not be easily answered by sighted adults. Han- dling such questions (e.g., When this building constructed? 1955) is critical they come closer to the natural distribution of human questions.
1 Introduction
The acquisition of knowledge occurs in the pre- training of large language models (Brown al., 2020; Chowdhery al., 2022), demonstrated as their emergent ability answer information- seeking questions in the open- sornd where the questioner does not have easy access to the infor- mation. While prior works have analyzed models' capabilities to answer textual information-seeking (or info-seeking) questions. much less is known for visual info-seeking questions. For example,
Work done when interned Google
'Our dataset is available at https:// open-vision- Languugge gww..b..tt........ wtw.chioo/oosseee io/infoseek/.
In this paper, we present INFOSEEK, a natural

---

VQA dataset that focuses on visual info-seeking questions. Different from previous VQA datasets, the testing subset of INFOSEEK collected in multiple stages from human annotators to evalu- ate VQA where the question can not be answered from only the visual content (see comparison of datasets in $ 2). In addition to this manually cu- rated test set, which enables realistic evaluation of info-seeking VQA, also join annotations from a recent visual entity recognition dataset (Hu et al., 2023) with the Wikidata database (Vrandečié and Krötzsch, 2014), and employ human annotators to write templates to semi- autitattomatiallly generate a large corpus of visual info-seeking QA pairs. Over million (image, question, answer) triplets are generated to support fine-tuning multimodal models for info-seeking VQA. We split data sure memorizing knowledge during fine-tuning is useless models either have to learn to use knowl- edge learned during pre-training or learn to retrieve knowledge from an external knowledge base.
Dataset
OK-VQA VQA
ViQuAE
INFOSEEK
PaLM (Q-only)
23.8
31.5
5.6
Current SotA
66.1
22.1
18.2
Require Knowledge*
29.2%
95.2%
95.6%
f 1% of questions that requine knowledge to answer. PaLM (Q-only): question-oely bascline using PaLM.
Table I: Comparison of INFOSEEK and prior KI-VQA benchmarks. Performances reported VQA score.
experiment ($5.2) further suggests that improv- ing visual entity recognition can drastically in- crease model's capability in answering visual info- sceking questions (from 18% to 45.6%), indicating a promising direction for future development.
The Need for a New Visual Information-ssekingg Benchmark
While there have been plenty of knowledge- intensive VQA (KI-VQA) benchmarks, we show that none of these meet the criteria to effectively evaluate info-seeking VQA. Early efforts in this area, such as KBQA (Wang et al., 2015) and FVQA (Wang et al., 2017), were based on domain- specific knowledge graphs, while recent datasets like OK-VQA (Marino al., 2019) and its vari- ants such as S3VQA (Jain et al., 2021) and A- OKVQA (Schwenk et al., 2022) have improved upon this foundation by incorporating an open- domain approach and highlighting common-sense knowledge. Among the existing benchmarks, K- VQA (Sanket Shah and Talukdar, 2019) and Vi- QuAE (Lerner al., 2022) the most relevant, but they have severe limitations in their question generation process, as discussed below.
Using INFOSEEK, we analyze the ability of state- of-the-art models to answer visual info-seeking questions. We found pre-trained vision-language models, such as models pre-trained end-to-end (e.g.. PaLI-X by Chen et al.), and models pre-trained with frozen LLM (e.g., BLIP2 by Li et al.), both struggle to answer info-seeking questions in zero- shot, though BLIP2 outperforms PaLI-X by a mar- gin. Surprisingly, after fine-tuning on our (large, semi-automatically curated) training set, PaLI-X yields a significant improvement and outperforms the fine-tuned BLIP2 models on queries that are unseen during fine-tuning. This suggests that while pre-trained PaLI-X has significant amount of knowledge, it requires small amount of fine- tuning data fully awaken its capabilities. Further- more, we show that INFOSEEK fine-tum ten-t models can even generalize to questions and entity types completely unseen during fine-tuning (c.g., art & fashion).
Information Seeking Intent. The evaluation of models' ability to answer info-seeking questions requires fine-grained knowledge, which a person is unlikely to know off the top of their head. How- ever, we found that 70.8% of OK- OK-VQA ques- tions? can be answered without the need to search engine, indicating the dataset primarily focuses on knowledge that is commonly known to people. Most OK-VQA questions are regard- ing coarse-grained knowledge that many people already know: What days might I most commonly go to this building? Sunday. One only needs to know the building type (e.g., Church) rather than the specific building (e.g., Dominus Flevit Church). This
When incorporating a visual entity recogni- tion component, and conditioning models on the Wikipedia articles of the relevant entities, we show that models accessing such knowledge base (With-KB) perform better overall than those that rely on knowledge learned during pre-training. However, end-to-end (No-KB) models were found better on certain classes of questions that require coarse-grained answers ( ("Which continent this building located on?"), even on tail entities. Our
"Studied with human on 500 random OK-VQA questions (see Appendix C.I)

---

makes it unsuitable for evaluating pre-trained mod- els on long-tailed knowledge, where these models have shown weaknesses (Kandpal et al., 2022).
Dataset
[1, A)
Len of Q/A
Entity
Ent. type
OK-VQA
14K
8.1/1.3
K-VQA
183K
10.1/1.6
18,880 2,397
ViQuAE
3.6K
12,4/1.7
980
Reliance on Visual Understanding. In contrast to OK-VQA, the ViQuAE dataset aims to test fine-
INFOSEEK
Wikidata
1.35M
8.9/1.5 7.8/2.3
11,481
2,739
grained knowledge of visual entities by pairing questions from TriviaQA (Joshi et al., 2017) with images. However, significant portion of the Vi- QuAE questions (e.g., "Who betrayed him for 30 pieces of silver?") can be answered without look- ing at the images, as the questions often reveal suffi- cient information to determine the answer. To quan- tify this observation, we present questions from the evaluation set to large language model, PaLM (540B) (Chowdhery et al., 2022). Results on the ViQuAE test set are shown in Table 1. Surprisingly, we find that PaLM can read questions and generate answers with 31.5% accuracy, outperforming the SOTA retrieval-based model (Lerner et al., 2022) (which has access to the image) on this dataset by 9.4%. Although PaLM a much larger model, this experiment illustrates that is possible to achieve very good performance on ViQuAE without using information from the image.
Human
8.9K
806
527
VQA does specify visuall entities.
K-VQA only covers entities from the human category.
Table 2: Statistics of INFOSEEK & KI-VQA datasets.
INFOSEEK Wikidata an automated dataset (1.3M) to cover diverse entities for large-scale training and evaluation purposes (see $ 3.2). We split the dataset to ensure memorizing the training set is useless, thereby emphasizing the importance of pre-training to acquire knowledge (see $ 3.3). Due to space limitations, we summarize the key essence in this section and defer details to the Appendix.
Image Sources for Diverse Entity Coverage. We sourced images from 9 image classification and retrieval datasets used in Hu et al., including land- marks (17%), animals (13%), food (5%), aireraft (3%), etc. We utilize their annotation, which links visual entities to their corresponding Wikipedia ar- ticles, construct our INFOSEEK dataset.
Entity Coverage. Current VQA datasets often cover a limited number of visual entity categories. For example, K- VQA only focuses on human sub- jects, while over 43% of questions in ViQuAE re- volve around human entities (see Table 2). Such limitations hinder the evaluation of a model's knowledge across various entity categories and may result in reduced task complexity, as the evaluation may be limited to mere facial recognition.
3.1 INFOSEEKJuman Natural Info-Seeking VQA Data Annotated by Humans
To ensure INFOSEEK questions rely visual un- derstanding and prevent models from taking short- cuts in the question without using the image, we employ a two-stage annotation approach inspired by TyDiQA (Clark et al., 2020). This makes it unlikely questioners will have prior knowledge of the answer like SQuAD (Rajpurkar et al., 2016), ensuring questions with info-seeking intents (Lee et al., 2019).
To address these limitations, we present INFOS- EEK (S 3),
benchmark for pre-trained multi- modal models visual info-seeking questions. Our work builds on top of a visual entity recogni- tion dataset, OVEN (Hu et al., 2023), which is de- signed to answer questions related to the identifica- tion of visual entities. We take visual info-seeking a step further by benchmarking info-seeking ques- tions about visual entities, which allows us to test the pre-training knowledge of models beyond sim- ply recognizing entity.
Question Writing. Annotators asked to write 3-5 questions about visual entity based on their own curiosity and information needs. To aid the question-writin process, they are prompted with visual entity images, a short description (15 words) about the entity, and a list of Wikipedia section titles. This ensures that the questions reflect a gen- uine interest in learning about important aspects of the entity without seeing the answer. A set of anno- tation rules is employed prevent trivial questions, such as questions about visual attributes.
3 INFOSEEK: A VQA Benchmark of Visual Information-ssekinng Questions
The INFOSEEK dataset consists of two compo- nents, (1) INFOSERK H collection of human- written visual info-secking questions (8.9K) to sim- ulate information seeking intent (see $ 3. 1); and (2)
Answer Labeling. For each entity, we randomly assign collected questions to a different group of an-

---

Which year was th's building constucted?
Modela with No KB
which builiig constructec??
Models Sub-lask ft: Visuel Entity Recogeition
End2End
Answer
Sub-tssk #t:
Language QA
Anawer
Prompt
BLIP2
1901
PaLM
1955
Read BiEPIIsA
Dominus Church
PaLi 1951
CLIP
FiD
1955
Models with No KB access
(b) Models With KB (Knowledge-base) information
Figure 2: Visual info-seeking models under the proposed No KB and With KB protocols. (a) End-to-end VQA models (such as PaLI (Chen et al., 2023b) or BLIP2 (Li et al., 2023b)) that directly predict the answer from looking the image and question; (b) Pipeline systems with access to knowledge base (e.g. Wikipedia), with the option link the queried subject to the Wikipedia use CLIP (Radford et al., 2021) and perform textual question-answent using PaLM (Chowdhery et al., 2022) or Fusion-in Decoder (FiD) (Izacard and Grave, 2020).
notators to label answers from Wikipedia. Annota- tors were shown the Wikipedia article of the entity and asked find concise answer to the ques- tion: a text span that is as short as possible while still forming a satisfactory answer. In addition, annotators categorize questions into three types: TIME (e.g., year), NUMERICAL (e.g., height) and STRING (c.g., location).
pairs. QA Pair Filtering and Subsampling, To ensure the questions diverse and the answers be referenced from Wikipedia, we filter out QA pairs when answers from Wikidata cannot be found in the Wikipedia article and subsample questions balance the distribution of entities and relations.
Finally, we construct (image, question, answer) (IQA) triples by assigning images for the annotated QA pair of a visual entity, followed by human veri- fication and clarification of the questions if multi- ple objects are presented in the image. Following TyDiQA (Clark et al., 2020), we measure the cor- rectness of annotations and take the high accuracy (95%) as evidence that the quality of the dataset is reliable for evaluating visual info-sceking models.
3.3 Evaluation of INFOSEEK
Dataset Split. We design the evaluation split to prevent overfitting to the training set and focus on evaluating the generalization ability of the pre- trained models. This includes the ability to answer questions of new entities and questions not seen during training. Particularly, we define evalua- tion splits: (1) UNSEEN ENTITY, where portion of entities are held out during training and only in- cluded the evaluation; (2) UNSEEN QUESTION, where we hold out a portion of the QA pairs of seen entities for evaluation.
3.2
INFOSEEKwaadd : Million Automated VQA Data from Wikipedia
Human annotation is valuable but costly for large- scale evaluation. We thus scale up the dataset using a semi-automated procedure, transforming knowledge triples in Wikidata (2022-10-03) to nat- ural language questions with numan-authore tem- plates, resulting in 1.3M examples over 1 11K visual entities covering 2.7K entity types (see Table 2).
Evaluation Metric. Three types of questions are evaluated differently: VQA accuracy (Goyal et al., 2017) for STRING and TIME; Relaxed Accu- racy (Methani et al., 2020) for NUMERICAL. We applied different relaxing strategies for each ques- tion type, and averaged the accuracy for each ques- tion. Finally, we calculate the accuracy for each data split (UNSEEN QUESTION and UNSEEN EN- TITY), and take the harmonic mean of them as the overall accuracy (see Appendix).
QA Generation. We convert knowledge triples (subj, relation, obj) in Wikidata to natural language question-anseee aaaswern-answerr pairs for selected list of 300 re- lations. For each relation, annotators write one two question templates, which contain place- holder for hypernym of the visual entity (e.g.. car) and placeholder for unit measurements (e.g.. inches) in numerical questions to avoid ambiguity. Finally, we construct the IQA triples by pairing images of a visual entity with corresponding QA
4 Protocols and Models for INFOSEEK
Motivated by previous research on text-based ques- tion benchmarks (Joshi et al., 2017; Roberts et al., 2020), we introduce two evaluation protocols, i.e,
Baasdd on manual inspection of 500 examples, we found this process rarely produces incorrect examples (11.2%).

---

Eval Protocol Training/Validation
Testing
Methods
Example Models
Knowledge Base
No-KB With-KB
(I.Q.A)
(I.Q]
End-to-end Model
PaLL BLIP2
(),
(L,Q)
Pipeline System
CLIP PaLM /FD
Wikipedia
Table 3: Two evaluation protocols of INFOSEEK. The key difference is whether auxiliary data for visual entity recognition and knowledge base is available at training. I: image, Q: question, A: answer, queried visual entity.
No KB and With KB, to evaluate models with dif- ferent information accessible from INFOSEEK. Ta- ble 3 and Figure 2 have provided a comparison for the two setups. This key design choice is made encourage models from different families to be compared with clear notion of what information was accessed. We note that the No KB protocol more challenging than the With KB protocol.
ing set questions grouped by question 4-gram. PaLM (Q-only) Model. To validate the impor- tance of visual content in INFOSEEK, we build question-only baseline with PaLM (540B) (Chowd- hery et al., 2022), using text question as the only input and with 5-shot in-context-learning.
BLIP2 & InstructBLIP. We utilize two pre- trained vision-language models, i.e. BLIP2 (Li et al., 2023b) and InstructBLIP (Dai et al., 2023). Both models share the same archi- tecture. which trains a Q-former Transformer that connects a frozen vision encoder (ViT-g/14) to a frozen instruction-tuned language model (Flan-T5ya (Chung et al., 2022)) to output text based on an input image and text. Particularly, Instruct- BLIP fine-tunes the BLIP2 model on 26 vision- language datasets (e.g.. VQAv2, OKVQA) with a text instruction prefix, and claimed to show improved zero-shot performance on unseen vision-language tasks. Following Li al. (2023b), we fine-tune the Q-former of both models using the INFOSEEK Wikidata: for improved performance.
The No-KB protocol. Models are tasked to di- rectly predict the answer by examining the image and question, similar traditional VQA systems. This requires the model to store world knowledge in its parameters for effective question answering. The research question focuses on how much knowl- edge can an end-to-end model memorize its pa- rameters during pre-training, and how well can it utilize this knowledge after fine-tuning" We use the standard VQA formatted data, i.e, (Image ()), Question(() Q), Answer(A)) triplets for training/ / val- idation, and (1, Q) for testing.
The With-KB protocol. The goal is to analyze headroom for improvement when a viable reason- ing chain is explicitly provided. Therefore, this protocol encourages an extra step of visual entity recognition, grounding the task on a knowledge base The VQA task is transformed into two-step pipeline, i.e, (1) visual entity recognition; and (2) language QA with entity information. We provide training signals to first recognize the queried visual entity and then leverage the information to query a large language model for answers, or identify rel- evant Wikipedia articles for extracting the answer. Specifically, we provide 100K Wikipedia KB (articles and infobox images) that includes visual entities from INFOSEEK and top frequent entities from Wikipedia. During training and validation, With KB protocol provides entity labels for each queried visual entity. During testing, the model is evaluated based on the (1, Q) pairs only.
PaLI-17B & PaLI-X. We experiment with two extra pre-trained vision-language models from the PaLI (Chen et al., 2023b,a) family given its SOTA performance. Particularly, we use PaLI-17B (ViT-e + mT5XXL (Xue et al., 2020)) and PaLI-X (ViT- 22B (Dehghani al., 2023) UL2-33B (Tay al., 2022)), which are pre-trained on WebLI (Chen et al., 2023b) with billion image-text pairs. Both models, which use non instruction-tuned language models, exhibit minimal zero-shot performance on INFOSEEK. Consequently, we fine-tune both mod- cls on the INFOSEEK Wikidata to improve their per- formance.
4.2 Models with KB Information
In this protocol, we explicitly model the path to answer info-secking questions with two decou- pled sub-tasks: (1) recognizing the visual entity grounded to the KB and (2) textual reasoning answer the question. A hidden benefit of such pipeline systems is improved interpretability, be- cause it is easier to locate the source of errors by
4.1 Models without KB Information
Random & Prior. Random answers sampled from the training set; The majority answer based on the question prior, which is calculated using the train-

---

INFOSEEK Wiadun
INFOSEEK
OK-VQA VQA+2
Model
LLM
Params
UNSEEN QUESTION
UNSEEN ENTITY
UNSEEN QUESTION
UNSEEN ENTITY
Overall
Overall
Accuracy Accuracy
Random
0.1
0.1
0.1
0.2
0.1
0.1
Prior
3.9
2.7
3.2
0.3
0.3
0.3
32.1
PaLM (Q-only)
PaLM
540B
5.1
3.7
4.3
4.8
6.6
5.6
23.8
43.0
BLJP2
Flan-T5xx
12B
14.5
13.3
13.9
10.0
8.9
9.4
54.7
82.3
InstructBLIP
Flan-T5xxt
12B
14.3
13.2
13.7
10.6
9.3
9.9
55.5
PaLI-17B
xxxxx
17B
20.7
16.0
18.1
13.3
5.9
8.2
64.8
84.6
PaLI-X
UL2g28
55B
23.5
20.8
22.1
12.9
9.3
10.8
66.1
86.1
F: Numbers adopted from Agrawal al.
Table 4: Results of No-KB models fine-tuned on INFOSEEK. Baselines including Random, Prior (majority answer with 4-gram question prior), and question-only model using PaLM (Q-only) with 5-shot prompting. VQA accuracy of models on OK-VQU VQA (Marino et al., 2019) and VQAv2 (Goyal et al., 2017) are for comparison.
diagnosing each sub-task component.
model, which reads information from a KB, understand the value of Wikipedia articles in the KB. Specifically, the FiD (Izacard and Grave, 2020) model employed, which takes N=100 re- trieved articles as input and generates an answer. The model is pre-trained with La (Raffel et al., 2020) backbone (660M) on Natural Ques- tions (K (Kwiatkowski et al., 2019) and fine-tuned INFOSEEK. During inference, retrieve the first 20 passages from Wikipedia for k=5 visual entities (from entity recognition) and feed 100 passages to FiD to generate the answer.
Sub-task #1: Visual Entity Recognition. We follow the entity recognition task defined in OVEN (Hu et al., 2023), and use an image and a text query (e.g., "What is this building?") as model inputs, and predict entities among 100K multi-modal Wikipedia entries. Particularly, employ the pre-trained CLIP (Radford al., 2021) model (ViT-L/14), as our visual entity recognition model, because its strong generalization capabil- ity. Specifically, we follow the CLIP2CLIP model described in Hu et al., to fine-tune CLIP to en- code multi-modal representations (image, question) from our dataset query, and (Wikipedia image, Wikipedia title) from the KB candidates. We then retrieve the top k=5 most similar entities based on weighted cosine similarity scores computed be- tween the query and candidates.
Experiments
5.1 Results for No-KB Models
Main results. Table presents the results of end- to-end models on INFOSERK. The best pre-trained model in this setting is PaLI-X, although the abso- lute number on the model's overall performance remains low. This is partially due to the fact that INFOSEEK questions often require identifying enti- ties and retrieving specific information relevant to the question, making it challenging task for end- to-end models. As PaLI-X is pre-trained on large corpus with more model parameters, it demon- strates better generalization ability on the UNSBEN ENTITY split compared to PaLI-17B. Meanwhile, there remains a noticeable gap in performance on the UNSEEN QUESTION and UNSEEN ENTITY splits, indicating that models struggle with gen- cralization to visual entities from the train- ing set. We also present models' results OK- VQA (Marino al., 2019) and VQAv2 (Goyal et al., 2017) for comparison and observe a dras- tie performance gap, emphasizing the difficulty visual info-seeking questions.
Sub-task #2: Language QA with LLM or KB Reader: Through visual entity recognition, we can now represent the queried visual information as its textual description. This enables to investigate the language reasoning component independently understand how much improvement a strong LLM or a KB reader can bring.
PaLM: Large Language Model. We use PaLM (540B) to investigate the amount of knowledge that can be memorized the model's parame- ters from pre-training on text corpora. Given a question and the queried entity name (from en- tity recognition), prompt PaLM predict the answer using 5-shot in-context examples with the prompt format: "question: This {entity) (question) answer:".
Fusion-in Decoder (FiD): KB Reader. We experiment with SOTA reureeedl-augmeetted

---

INFOSEEK
INFOSBEK
20
Zero-shot
Accuracy
0 Fine-tuned
10
what year was this painting created? 7-178: PaLI-X: 1884 /
which year was this brand established?
PaLI-17B
PaLI-X
Bl BLIP2
InstructBLIP
PaLI-17B
PaLL-X
Bl BLIP2
InstructBLIP
1-178::
PaL1-X: 1854
BLIP2: 1887
BLIP2: 1854
Figure 4: Predictions on out-of out-of-domain visual entities (art & fashion) collected from real-world images by authors, using INFOSEEK fine-tuned models.
Figure Zero-shot fine-tuned performances on INFOSERK. Fine-tuning on INFOSERK elicits knowl- edge from PaLI models to answer ine-grained visual info-secking questions.
Fine-tuning elicits knowledge from the model. To demonstrate the value of INFOSEEK training data, we report the zero-shot performance of mod- els in Figure 3. Specifically, we find that with- out fine-tuning, both PaLI models produce neg- ligible overall performance, which is significantly worse than the fine-tuned counterpart. This pro- vides evidence to support the hypothesis that fine- tuning has helped elicit knowledge from the pre- trained PaI.I models. On the other hand, BLIP2 and InstructBLIP show compelling zero-shot per- formance on INFOSEEK as they adopt a frozen instruction fine-tuned LLM (i.e, Flan-T5) and In- structBI.IP is further instruction-tuner on col- lection of VQA benchmarks. The performance of BLIP2 models is further improved after fine-tuning INFOSEEK with a small number of steps, show- ing strong generalization results the Human split. Figure 10, we present examples of BLIP2 pre- dicting the "country location" an unseen entity (Leemberd) and show the accuracy was improved from 18% to 92% after fine-tuning, despite not see- ing this entity in the training set. Finally, we con- ducted real-world evaluation on out-of-domain images unavailable from the Internet (not from any models' pre-training data). Particularly, we evalu- ate fine-tuned PaLII with 90 questions on 30 images captured by the authors, on visual entities outside the INFOSEEK training corpus. As result, PaLI- 17B and PaLI-X answered 22.2% and 38.9% of questions correctly. Figure 4 presents examples of PaLI and BLIP2 predictions on two out-of-domain entities (artwork and fashion product).
Q: Which body of water is this mountain located A: Lake Como
Q: Who designed this bridge?
A: Thomas Telford
BLIP2ce-soe) lake Instructal (P lake
BLIP2 john
Instructel 1P (re-stocy archiledt
Figure 5: InstructBLIP (e-shot) makes less fine- grained predictions compared to initial model (BLIP2), after instruction-tuned prior VQA datasets.
ered for future model development, We found InstructBLIPg--se performs significantly worse than its initial checkpoint, BLIP2 (7.4 vs 11.3 on InfoSeek nikisess) which contradicts the superior zero-shot performances of InstructBLIP in Dai al. (2023). We conduct manual analysis and detect a common error made by InstructB is its pref- erence for generating coarse- graiseer predictions compared to BLIP2 (e.g., archi tect person's s name). This leads to performance drop IN- FOSEEK, which emphasizes fine-grained answers (see Figure 5). We hypothesize that this can be attributed to the instruction tuning datasets used for InstructBLIP (e.g., VQAv2 and OK-VQA), which share a less fine- nee-grained answer distribution. Fortunately, fine-tuning on INFOSEEK Wikidata helps close the gap.
5.2 Results for With-KB Models
Models with KB access perform better. Table 5 presents the results for pipeline models with ac- cess knowledge base (KB) information, along with the best results from the No-KB setting for reference. Notably, the pipeline models outper- form the best No-KB models on the challenging INFOSEEK Human split significantly. This highlights the pipeline systems' ability to answer visual info-
Why does instruction-tuned BLIP2 obtain worse zero-shot INFOSEEK results? One surpris- ing finding from Figure 3 caught our attention and reveals an important criterion to be consid-

---

INFOSBEK
INFOSEEK
ENTITY Accuney
brand..?
Model
Whiddm
Human
Which country.?
Best No-KB
22.1
10.8
Which continent..?
With-KB
With-KB Setting
No-KB
CLIP PaLM
20.1
15.2
What the mountain range..?
22.2
CLIP - FID
19.3
18.2
20
40
80
Oracle + FID
52.0
45.6
100
Accuracy %
Table Results of With-KB setting. CLIP +
Figure 6: No-KB (PaLI-178) outperforms With-KB (CLIP->FID) models questions that query less fine- grained attributes.
PaLM/FID: a two-stage pipeline system (visual entity recognition - text QA). PaLM: 5-shot prompting. FID: Fusion-in Decoder to read from KB using targe- Ora- cle: an artificial upper-
upper-bound bound using oracle entities.
to-end models such as PaLI, have short barrel on fine-grained knowledge-intensing questions (i.e, TIME and N NUMERICAL). can perform well on other questions, which are more about querying at- tributes or resolving relations between entities (see Figure 6). Comparing With KB models, PaLM and FiD perform on par with each other on this au- tomated evaluation data. However, when evaluated on the natural info-seeking human queries, FiD has a better generalization, outperforming PaLM on TIME (21.5 vs 14.6) and NUMERICAL (25.6 vs 21.3) questions from INFOSEEK Human signif- icantly. One possible reason is that natural info- sceking questions written by people focus more on very fine-grained information, which is rare and hard to memorize for PaLM. In contrast, FiD can leverage Wikipedia articles to predict answers. Finally, we analyze the performance of different models according to the visual entity popularity and found unique advantages end-to-end models (see Appendix).
TIME (Ace.)
NUMERICAL
STRING (Acc.)
Model
(Relaxed Acc.)
No-KB Setting
PaLM (Q-only)
11.4
InStructBLIP
7.9
BLIP2
6,9
PaLI-17B
3.8
18.4
PaLI-X
7.7
16.1
30.0
With-KB Setting
CLIP Pal.M
12.5
27.7
21.7
CLIP FiD
12.3
23.4
23.9
Table 6: Results w.r.t. each question types on the INFOSEEKwaidn val set unseen question split, show- ing big headroom for improvements TIME and NUMERICAL for all end-to-end models.
seeking questions by effectively utilizing visual recognition and language reasoning, specifically using the names of visual entities to convey infor- mation across modalities. When comparing the Language QA models, we observe that the FiD model, which reads the Wikipedia article, achieves the highest generalization performance on INFOS- EEK Human by a significant margin. This suggests that access to relevant text content plays a crucial role in answering visual info-secking questions.
Performance on Head vs. Tail entities. Al- though pipeline models with KB access are overall stronger, surprisingly, we observe that end-to-end models have unique advantage for info-seeking VQA, particularly the tail entities. Figure 7 presents a comparison of models, with group-wise performances on Wikipedia entities that are least popular (less monthly page views) to most popu- lar (more monthly page views). The histogram is generated based on the average monthly Wikipedia pageviews in 2022, following (Mallen et al., 2022). Surprisingly, the results show that PaLI-17B out- performs the pipeline systems by large margin on the tail entities, particularly for questions lated to geographical information. We show some qualitative examples in Figure 9, for entities from baskets different monthly page views. This sug- gests that there are many different routes to answer
Large headroom for improvement. Table demonstrates an artificial upper-bound (Oracle - FiD) on INFOSEEK, indicating substantial room for performance improvement nition model were available. By simulating the visual entity recognition's s accuracy improvement (from 22% using CLIP to 100%), the INFOSEEK accu- racy can be improved from ~20% to 50%% within the same FiD model.
oracle entity recog-
Analysis on each question type. Table shows a breakdown of results under different question types, evaluated on INFOSEEK Wikidata- Comparing No KB and With KB models, we found that end-

---

40
ploy model (Izacard and Grave, 2020) per- form language QA (Gui al., 2022; Lin al., 2022). Other approaches transform the image into a text caption and use an LLM (Brown et al., 2020; Chowdhery et al., 2022) to answer questions (Yang et al., 2022; Hu et al., 2022a). We utilize both approaches to study the ceiling for improvement on INFOSEEK with the OVEN model (Hu et al., 2023).
D7 JICLII M CLIP-FFiD
Accuracy %
20
10
100 500 IK 2.5K 5K 10K 25K 50K 1OOK IM
Average Monthly View of Entity
Another concurrent work (Mensink et al., 2023) investigates similar challenges but emphasizes seal- ability and relies on model-generated annotations, as opposed to our aananna-d seekin queries.
Figure 7: INFOSEEK results w.r.t. visual entities of different popularity. End-to-end model outper- forms pipeline systems on tail entities (low momtlyy pageviews) but overturned more popular entities (high monthly pageviews).
7 Conclusion
visual info-seeking questions and that pipeline sys- tems that rely on an explicit decomposition of the VQA task may be redundant and susceptible to сπrоr propagation from the entity linking stage. Whereas for end-to-end models such PaLl, it is flexible to decide which route of reasoning is more appropriate answer given question. For example, one can answer geographical questions without knowing the identity of the visual entity, if other relevant visual clues are presented. Mean- while, on the more popular head visual entities, a clear trend emerged showing that pipeline systems outperform end-to-end PaLI by a big margin.
We introduced INFOSEEK, a large-scale VQA dataset that focuses on answering visual informa- tion seeking questions. With INFOSEEK, we found that current state-of-the-art pre-trained visual- language models struggle to answer visual info- seeking questions requiring fine-graines knowl- edge, such as questions about time and numerical information of visual entity. Our analysis using pipeline systems, which ground visual entities an external knowledge base, suggests that incorpo- rating fine- aue-grained knowledge into the pre-training process holds significant potential to improve end- to-end pre-training models.
6 Related Work
8 Limitation
Pre-trained Vision Language Models. There has been significant growth in the development of vision-language models pre-trained on large-scale image-text datasets (Lu et al., 2022; Bao et al., 2021; Wang et al., 2022; Zhou et al., 2020; Radford et al., 2021). One line of research aims to augment pre-trained language model with visual modality by learning a mapping from an external visual en- coder the frozen large language model (Alayrac al., 2022; Li et al., 2023b; Koh et al., 2023), to fully leverage textual knowledge from the language model (Xu et al., 2023; Dai et al., 2023; Liu et al., 2023; Zhu et al., 2023; Ye et al., 2023).
INFOSEEK is limited to English language and fu- ture research could expand it to a multilingual set- ting, leveraging articles in Wikipedia supported in other languages. While the primary focus of this work is on knowledge derived from Wikipedia, future investigations could explore extensions to other domains, such medical information, and artwork, and incorporate emerging updates in Wikipedia (Iv et al., 2022).
Acknowledgement
We thank Jialin Wu, Luowei Zhou for reviewing an early version of this paper. We thank Xi Chen for providing different variants of PaLI pre-trained checkpoints. We also thank Radu Soricut, Anelia Angelova, Fei Sha, Andre Araujo, Vittorio Ferrari, Wei Xu, Kartik Goyal for valuable discussions and feedback on the project. Yang Chen is partially funded by the NSF (ISS-2052498.
Knowledge-based VQA Models. Various approaches have been proposed to address knowledge-based VQA tasks (Marino et al., 2019) by incorporating external knowledge into vision- language models. One approach is to retrieve infor- mation from an external KB (Marino et al., 2021; Hu et al., 2022b; Wu and Mooney, 2022) and em-

---

References
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. In- structblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint 2305:2300006500
Aishwarya Agrawal, Dhruv Batra, Devi Parikh, and Aniruddha Kembhavi. 2018. Don't just assume; look and answer: Overcoming priors for visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4971-4980.
Mostafa Dehghami, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. 2023. Scaling vision trans- formers to 22 billion parameters. arXiv preprint arXiv:2302.05442.
Jean- Sann-tatistt Alayrac. Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, al. 2022. Flamingo: a visual language model for few-shot learning. Conference on Neural Informa- tion Processing Systems.
Gerry. 2021. 100 sports image classification.
Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. 2021. Beit: Bert pre-training of image transformers. International Conference on Learning Representa- tions.
Yash Goyal, Tejas Khot, Douglas Summers-Stay. Dhruv Batra, and Devi Parikh. 2017. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Computer Vision and Pattern Recognition.
Lukas Bossard, Matthieu Guillaumin, and Lue Van Gool. 2014. wood-101-mmming discriminative components with random forests. In European Conference on Computer Vision. Springer.
Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Haupt- mann, Yonatan Bisk, and Jianfeng Gao. 2022. KAT: A knowledge augmented transformer for vision-and- language. Proceedings of the Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technolo- gies.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Conference on Neural Information Process- ing Systems.
Danna Gurari, Qing Li, Abigale J. Stangl. Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey Bigham. 2018. Vizwiz Grand Challenge: Answering visual questions from blind people. In Computer Vision and Pattern Recognition.
Xi Chen, Josip Djolonga, Piotr Padlewski, Basil Mustafa, Soravit Changpinyo. Jialin Wu, Car- los Riquelme Ruiz, Sebastian Goodman, Xiao Wang. Yi Tay, et al. 2023a. Pali-x: On scaling up a multi- lingual vision and language model. arXiv preprint arXiv:2305. 185665
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.0965.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergio- vanni, Piotr Padlewski, Daniel Salz, Sehastian Good- man, Adam Gryener. Basil Mustafa, Lucas Beyer, al. 2023b. PaLi: A jointly-scaled multilingual language-image model. International Conference on Learning Representations.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandel- wal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming Wei Chang. 2023. Oppen-domain visual entity recognition: Towards recognizing millions of wikipedia entities. arXiv preprint arXiv:2302.1 11154.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts. Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. 2022a. Promptcap: Prompt-guided task-aware image captioning. arXiv preprint arXiv:2211.0000 1.09699.
Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai- Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2022b. Reveal: Retrieval- augmented vinual-language pre-training with multi- source multimodal knowledge memory. arXiv preprint 222.0222...
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetunee language models. arXiv preprint arxiv:22 11416.
Robert Iv, Alexandre Passos, Sameer Singh, and Ming- Wei Chang. 2022. FRUIT: Faithfully reflecting up- dated information in text. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Seattle, United States. Asso- ciation for Computational Linguistics.
Jonathan H Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020. TyDi QA: A benchmark for information-seeke secking question answering in ty- pologically diverse languages. Transactions of the Association for Computational Linguistics.

---

Gautier Izacard and Edouard Grave. 2020. Leveraging passage retrieval with generative models for open domain question answering. Proceedings of the Eu- ropean Chapter of the Association for Computational Linguistics.
Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. Blip-2: Bootstrapping language-img -mage pre- training with frozen image encoders and large lan- guage models. International Conference on Machine Learning.
Aman Jain, Mayank Kothyari, Vishwajeet Kumar, Preethi Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. 2021. Select. substitute, search: A new benchmark for showledgg-aaggmented visual ques- tion answering. In Proceedings of the 44th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 2491- 2498.
Yuanze Lin, Yujia Xie. Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. 2022. REVIVE: Re- gional visual representation matters in knowledge- based visual question answering. Conference on Neural Information Processing Systems.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning. arXiv preprint arXiv:2304.08485.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyrr. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Annual Meeting of the Association for Com- putational Linguistics.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Unified- io: A unified model for vision, language, and multi- modal tasks. arXiv preprint (1X.Xx:220660990
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. 2013. Fine-grained visual classification of aircraft. Technical report.
Nikhil Kandpal, Haikang Deng. Adam Roberts, Eric Wallace, and Colin Raffel. 2022. Large language models struggle to learn long-tail knowledge. arXiv preprint (tdi::2211.8411111
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das. Hannaneh Hajishirzi, and Daniel Khashabi. 2022. When not to trust language models: Inves- tigating effectiveness and limitations of paramet- ric and non-parametric memories. arXiv preprint arXiv:2212. 10511.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. 2023. Grounding language models to images for multimodal generation. International Conference on Machine Learning.
Jonathan Krause, Michael Stark, Jia Deng. and Li Fei- Fei. 2013. 3d object representations for fine-grained categorization. In Proceedings of the IEEE interna- tional conference on computer vision workshops.
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. 2021. Krisp: Inte- grating implicit and symbolic knowledge for open- domain knowledge-based vqa. In Computer Vision and Pattern Recognition.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin. Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. OK-V -QA: A visual question answering benchmark requiring external knowledge. In Computer Vision and Pattern Recog- nition.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq and Enamul Hoque. 2022. ChartQA: A benchmark for question answering about charts with visual and logical reasoning, Annual Meeting of the Association for Computational Linguistics (Findings).
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Annual Meeting of the Association for Computational Linguistics.
Thomas Mensink, Jasper Uijlings, Lluis Castrejon, Arushi Goel, Felipe Cadar, Howard Zhou, Fei Sha, André Araujo, and Vittorio Ferrari. 2023. Encyclo- pedic vqa: Visual questions about detailed prop- erties of fine-grained categories. arXiv preprint arXiv:230m 20...02022222 09224.
Paul Lerner, Olivier Ferret, Camille Guinaudeau, Hervé
Le Borgne, Romaric Besançon, Jose G. Moreno, and Jesús Lovón Melgarejo. 2022. Viquae, a dataset for knowledge- -bsee visual question answering about named entities. In Proceedings of the 45th Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. 2020. PloiQA: Reasoning over sci- entific plots. In Proceedings of the IEEE/CVF Win- ter Conference on Applications of Computer Vision, pages 1527-1536.
Dongxu Li, Jumnan Li, Hung Le, Guangsen Wang, Sil- vio Savarese, and Steven C.H. Hoi. 2023a. LAVIS: one-stop library for language- aanguage-vision intelligence. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume System Demonstrations), Association for Compu- tational Linguistics.
Maria-Elena Nilsback and Andrew Zisserman. 2008 Automated flower classification over a large num- ber of classes. In 2008 Sixth Indian Conference on Computer Vision Graphics & Image Processing.

---

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishkin, Jack Clark, al. 2021. Learning transferable visual models from natural language supervision. In International Con- ference on Machine Learning.
Peng Wang. Qi Wu, Chunhua Shen. Anton van den Hen- gel, and Anthony Dick. 2015. Explicit knowledge- based reasoning for visual question answering. arXiv preprint arXiv: 151110107000
Peng Wang. An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. Unifying architectures, tasks, and modalities through a simple sequence-to to- sequence learning framework. International Confer- ence on Machine Learning.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter Liu. 2020. Exploring the limits of transfer learning with unified text-to-text transformer. JMLR.
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. 2020. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and retrieval. In Computer Vision and Pattern Recogni- tion.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. Proceedings of Emperical Methods in Natural Language Processing.
Jialin Wu and Raymond J Mooney. 2022. Entity- focused dense passage retrieval for outside- knowledge visual question answering. Proceedings of Emperical Methods in Natural Language Process- ing.
Tal Ridnik, Emanuel Ben- Asaf Noy, and Lihi Manor. 2021. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104. 10972.
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the parame- ters of language model? CoRR, abs/2002.082 009900....9......
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. 2010. Sun database: Large-scale scene recognition from abbey to 200. In Computer Vision and Puttern Recognition.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, An- drej Karpathy, Aditya Khosla, Michael Bernstein, al. 2015. Imagenet large scale visual recognition challenge. International journal of computer vision, 115:211-252.
Zhiyang Xu, Ying Shen, and Lifu Huang. 2023, Multi- instruct: Improving multi-modal zero-shot learning via instruction tuning. Annual Meeting of the Associ- ation for Computational Linguistics.
Naganand Yadati Sanket Shah, Anand Mishra and Partha Pratim Talukdar. 2019. KVQA: Knowledge- aware visual question answering.
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. m5: A massively multilingual pre-trained text-to-text transformer. Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. urXiv preprint arXiv22206.01718.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022. An empirical study of GPT-3 for few-shot knowledge- based VQA. In Proceedings of the AAAI Conference on Artificial Intelligecce
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Gar- cia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. 2022. Unify- ing language learning paradigms. arXiv preprint 22Xiv:2205.05131.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, An- wen Hu, Pengcheng Shi, Yaya Shi, et al. 2023, mplug-owl: Modularization empowers large lan- guage models with multimodality. arXiv preprint arXiv:2304.1 14178.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. 2018 The inaturalist species classification and detection dataset. In Com- puler Vision and Pattern Recognition.
Victor Zhong, Weijia Shi, Wen-tau Yih, and Luke Zettle- moyer. 2022. RoMQA: A benchmark for robust, multi-evidence, multi-answer question answering. arXiv preprint 2rXiv:2110.1333.
Denny Vrandežié and Markus Krötzsch. 2014. Wiki- data: a free collaborative knowledgebase. Communi- cations of the ACM.
Luowei Zhou, Hamid Palangi. Lei Zhang, Houdong Hu. Jason Corso, and Jianfeng Gao. 2020. Unified vision- language pre-training for image captioning and VQA. In Proceedings of the AAAI Conference on Artificial Intelligence.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. 2017. FVQA: Fact-based visual question answering. IEEE transactions on pattern analysis and machine intelligence.

---

Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint 23X4v:2304. 10592.

---

QUESTION/ENIIY
PUNSEEN
TTual
TSM//NUH//SRR
Questina Type
dEniity
Correct
Incorrect
Triin
+/+
934,048
4,42004 75.2%
5,549
Percentage
95%
5%
Val
18 6565555,,,0 964
73,620
4.6 21, 13.87
1,794
9R.00 /340,079
26.8/
Table 8: Expert judgments of answer accuracy based on
sample of 200 examples from INFOSEEK Human-
Table 7: INFOSEEK Dataset statistics. Average ques- tion per image rate is 1.4 and 1.0 for Wikidata and Human split, respectively.
asking visual attributes (such as color), writing questions by rephrasing the description, copying entity names and section titles into the question, and avoiding writing ambiguous questions.
A Details of the Dataset.
In this section, we provide more details of the hu- man annotation quality control and automatic data generation process. We summarize the statistics INFOSEEK Table and show question prefix distribution in Figure 11 and entity distribution in Figure 12.
Stage 2 (Answer Labeling): As shown at the bottom of Figure 13, annotators are present with info-seeking questions to the entity collected from Stage 1 and Wikipedia link of the entity. For each question, annotators asked to find short span of answers (less than 10 words) from the Wikipedia page. They are asked to answer two questions: (1) "Can you derive the answer from the given Wikipedia page?" and (2) "What is the type of this question?" and select from three options (TIME, NUMERICAL, OTHERS). For each answer, they will then fill in the answer box (TIME: [year, month, day], NUMERICAL: [min, max, unit], OTHERS: [string]) and copy paste short sentence from Wikipedia that contains the answer to the evidence section. We decided to exclude questions without answer spans from Wikipedia following TyDiQA-GoldP as the dataset is already hard enough and reserve these questions for future work.
A.1 Human Annotation Quality Control
Instruction and Training. We hire 30 full-time in-house annotators to collect questions and an- swers in INFOSEEKHuan. Annotators are native English speakers in the U.S. and are aware of the purpose of the collected data. To ensure the qual- ity of annotations in the INFOSEEKHunan dataset, a comprehensive training process was designed and implemented for our annotators. This process involved pilot study, in which annotators read the instructions and annotated few sample ex- amples, followed by tutorial session and quiz. The tutorial was conducted through an online video session and provided a comprehensive overview the instructions while addressing common mis- takes identified in the pilot study. Only annotators who passed the quiz were selected to work on the main task, with 30 annotators completing the train- ing. We hire annotators at $17.8 per hour, which is higher than the minimum wage in the U.S., to fairly compensate annotators for their time and ef- fort. The average completion time for stages one and two of the annotation task was 12 and min- utes, respectively. A screenshot of the annotation interface provided in Figure 13.
Expert Feedback and Correction. Expert annota- tors provided regular feedback during annotation and conducted thorough post-annotation verifica- tion. The data was split into three batches, with annotators flagged and provided feedback for those who consistently made similar mistakes. After the completion of stage 1, questions that revealed the entity name, asked about the color shape of an object, or were binary were automatically re- jected. After stage 2, three expert annotators re- viewed and processed the question-answer pairs, removing unqualified pairs and verifying the an- swer span from the annotated evidence sentence. Rejected pairs may have included questions that were not answered by the annotated answer were too general and resulted in an ambiguous answer. The expert annotators also corrected the question type annotation and edited the answer span into the correct format, such as adding units for numeri- cal questions or shortening long answer spans that exceeded ten tokens. Finally, the expert annota-
Annotation Procedure
Stage 1 (Question Writing): As shown in Figure 13 (Top), annotators are shown with images of a visual entity the left-hand side with short description of the entity from Wikipedia below. On the right, we show list of Wikipedia section titles of the entity and ask annotators to write relevant questions next to the section title. We prevent annotators from asking binary questions,

---

Q: Who is the founder of this sport?
Q: What is the length of this bird in centimetre?
Q: Who is the inventor of this object? A: James Gregory
Q: What the country of origin of this animal?
A: Kanō Jigoró
A: 120-170
A: Rhodesia
In which year did this building officially open?
Q: Which year was this food invented? A: 1935
Q: What is the highest note this item can play? A: C8
Q: Where is this plant na- A: Ecuador
A: 1930
What is the cruise Which country is this speed this aircraft (kilo- meter per hour)?
Q: How heavy does this bird typically grow up in terms gram? A: 4.3
Q: Who is the structure engineer of this bridge?
animal endemic A: Madagascar
A: Hellmut Homberg
828
Q: What country does this mountain belong to?
Q: What is the area in square kilometers of this lake? A: 29.8
Q: What is this fish named after? A: lemon
Q: What is the closest par- ent taxonomy of this in- sect? A: Asterocampa
A: North Macedonia
Q: What the basionym this plant?
Q: What is the manufac- turer of this vehicle?
Q: Who does this place In which year was this commemorate?
building built?
A: Agaricus sect. Lepiota
A: Suzuki
A: Mohammad Jinnah 1661
Figure Random examples from the training set INFOSEEK Wikidata-

---

Endty
Head
Q: Which rogion this antmol U.S.A
What courtyy soos this building Bradi
O: What was the care thes type alioot Bo0k
What the alflude of this buildinng
1981 September End-tb-end (Palli: 1 22 X
2630 End-lo-end (Pak.)) 3000 Pipeline (Retrievaly 2400
End-fo-end (PaLII: U.S.A Pipeiino (Rootevvll) Brozl X
End-lo-end (Pal.) Brott o
Pipeline (Reerevall; United Kirgdon
Pipeline (Rethivall) 1981 September
Figure 9: Examples of predictions of PaLI-17B and CLIP-FII INFOSEEK (left to right shows tail to head entities).
Training Set Entities in Armenia
Validation Unseen Entityy Amberd (Q457057) Willpedia Google search:
BLIP2 Predictions:
country does this building belong to?
A: Armenia
Accuracy
BLIP 2/11 BLPP: 10/11
X
Figure 10: Examples of predictions of (o-hot) BLIP2 on INFOSEEK (Entity: Q457057). Fine-tuning improves the accuracy from 2/11 to 10/11, despite it being UNSHEN ENTITY (not in the training set). We show training set entities that are located in Armenia and images Amberd on the internet.
Figure 11: Question prefix distribution in INFOSBEK Wikiddra (left) and INFOSEEK Human (right).

---

used fuzzy string matching with substring ra- tio greater than 0.9 if an exact match could not be found. For time questions, we applied an exact match to verify the year, month, and date. In some cases, the year of construction of building varied by a year, so we allowed +/- year deviation for the time question. For numerical questions, we used exact matching verify the numbers in the article. However, in many cases, the units were different (meters or inches), range with min- imum and maximum was given. We used regular expressions to extract the number or range from the Wikipedia article and filter out the QA pairs if it is counted as incorrect based on the "Relaxed accuracy" in Section 3.3 in the main text. Based on manual analysis of 200 randomly sampled QA pairs, found that 97% of the answers of the IN- FOSEEK Wikidata could be found in the Wikipedia article.
2026
wile
Fri
Figure 12: Distribution of the entities in INFOSEEK (Grouped by their super category).
tors reviewed the sueeeeuuuusottnsweerr answer triples to reject bad images or clarify the question when multiple objects were present in the image. For specified when multiple buildings were present the image. On average, took 1.5 hours verify 1000 triples, as the majority images contained a single object.
example, a building
Subsampling Questions. In order to achieve a more diverse set of questions INFOŠEEK, we ap- plied a subsampling method to address the skewed distribution of crowd-sourced knowledge triples in Wikidata. The method followed the approach used in Zhong et al. (2022). This involved defin- ing P(r, c) as the percentage of triples that con- tain the relation r and the subject entity's is cate- gory as C. The c'(r,c) culated as the average probability a relation- category pair and Image-Q-A Q triples were removed with increasing likelihood based on the probability 1-mmm((1, (fr,ce)) /ere/)(//)))))))))ee))))) 1/2, Additionally, the same subsampling method was applied to bal- ance the answer distribution for each relation. This resulted in the question prior baseline achieving relatively low score (3.2) in INFOSEEK Wikidata as shown in Table in the main text.
Following TyDiQA (Clark et al., 2020), we ama- lyze the degree to which the annotations are correct instead of the inter-annotator agreement since the question may have multiple correct answers. In Table 8, human experts carefully judged sam- ple of 200 examples from INFOSEEK Human split. For each example, the expert reads through the Wikipedia page of the queried visual entity and finds the answer to the question. They then indi- cate whether the annotated answer is correct. We take the high accuracy (95%) evidence that the quality of the dataset offers valuable and reliable signal for evaluating visual info-secking models.
1/1(r, c)l was cal-
A.2 Filtering and Subsampling
A.3 Evaluation Metric.
Filtering. To test the models' ability answer visual aeformation-seeking questions that require fine-gran fine-grained knowledge, which can be learned from the pre-training corpus such as Wikipedia, we need to verify the consistency of answers between Wiki- data and Wikipedia. Given that Wikidata and Wikipedia are crowd-sourced independently, some QA pairs created from Wikidata may not be present in the Wikipedia article or may have different an- swers. Therefore, we filtered out QA pairs where the answer could not be found in the Wikipedia article of the entity. We performed an exact string match to verify answers for string questions and
There three types of questions, i.e, STRING, TIME, and NUMERICAL, which are evaluated differently. Particularly, we adopt the VQA accuracy (Goyal et al., 2017; Marino et al.. 2019) against multiple references for STRING and TIME questions, and utilize Relaxed Accu- racy (Methani et al., 2020; Masry et al., 2022) for NUMERICAL questions. For STRING questions, use the alias of answers from Wikidata as multiple references for INFOSEEK Wikidata (#avg = 4.5), and the human-annotated multiple
"SequenceMatcher from di diffiib library

---

Bndrutienn Aaczer sticke cancias
bue 4ll Bec't an Bibuaats bex
se auteticl lu
Bow di infermation
Galakaa
Puffy Omagge poatrd
Wikyeffe ban www.eitttiiiina aostuurrr ehCCCopp Auant pualect
Casyae dorke the cavurr Bran vdlgeela pigh?
parge
Toar
Manit::
A/l
Deieeso:: Lo tim bos trory para the ainarom rved derve anoverr
Figure 13: Annotation Interface Stage (Top) and Stage (Bottom).
references for INFOSEEK Human (#avg = 2.4).
Exact Match: Correct the prediction matches any one of the references exactly.
Exact Match: Correct if the prediction matches any of the references exactly.
mrediction=11911 references=="119900 "1991", "1992"])
prediction="USA"" references=""USA"" "U.S.", "United States America",
prediction="199 6 11", refcrences=["11911 11", "1991 June 11", "11 June 1991", +
For TIME questions, the answer references count for different date formats of year/month/day, Meanwhile, we perform relaxed match (with one-year enror tolerance) to measure the model's prediction, because it is quite often that historical events are only associated with estimated time.
For NUMERICAL questions, the exact match would not be able to handle the case where range (e.g. a pair of minimum and maximum values) is provided as annotated ground truth. To account for this, we make slight modification to the Relaxed Accuracy with 10% tolerance range.

---

Relaxed Accuracy; correct if the prediction is in the reference range the prediction range overlaps with the reference range of more than 50% !) ref min < pred ref_ sef_max: 2) loU[[pred min, pred max]. [ref min, ref max]) 2 50%
beam search decoding with a beam size 5. OFA achieves 11.7 and 4.0 on INFOSEEK Wikidata and Human split, respectively.
mPLUG-owl. We fine-tuned mPLUG-owl (Ye et al., 2023) for 10k steps with learning rate of 2e-4 and batch size of 1 using LoRA (Hu et al., 2021). mPLUG-owl achieves 7.7 on INFOSEEK
reference= 10
reference= 19, 11] 10% tolerance
prediction= 10, reference= [9, 11] +
prediction= [5, 6]. reference= 11]
A single value prediction is counted correct if it falls within the answer range, and a range prediction is correct if the intersection- -ofonn-ooooooon between the prediction and answer greater than or equal to 50%. Finally, we calculate the accuracy for each data split (UNSEEN QUESTION and UNSEEN ENTITY), and take the harmonic mean of them as the overall accuracy.
Human split.
PaLM(Q--nnyy. We use PaLM 540B (Chowdhery et al., 2022) in-context learning under the 5-shot setting with the following prompt:
Please answer the following question. question: [Question_1] ). answer: { Ansser-]] -1
question: [Question_i] _.). answer:
A.4 Image Sources.
B.2
With-KB Models
Image Recognition (or Retrieval) Datasets: Ik-P (Russakovsky et al., 2015; Ridnik et al., 2021), INaturalis20117 (Van Horn et al., 2018), Cars196 (Krause et al., 2013), SUN397 (Xiao et al., 2010), Food101 (Bossard et al., 2014), Sports100 (Gerry, 2021), Air- craft (Maji al., 2013), Oxford Flower (Nilsback and Zisserman, 2008), Google Landmarks v2 (Weyand et al., 2020).
Pull
PaLi-X
(Intrac)/LLP)
ORA
FID
Optimizer steps
Adafoctor
Adatactor
Adam
Adacaator
128
10k
800
400
21k
200
LR
lo-4
1c44
So-5
5o-5
2:-4
LR schedoler
conetum
polynomial drtpy comsbal
Warnup
1000
1000
1000
Image
224
224
224
480
VITe
WiTgg
backbone
mT15XL
Flan-T5xXL
BAXT
T5age 0.45
aPanms
171
35B
12.18
0.41
Compusing
32 TPC
64 TTC
A40
8 A100
64 TPU,
Time
hours
bour
1 hour
48 bouss
hour
B Implementation details of the baseline systems
Table 9: Hyperparameters for fine-tuning models on INFOSEEK.
this section, we provide complete implemen- tation details of baseline models for the INFOS- EEK task. We summarize hyperparameters for fine- tuming in Table 9.
PaLM. We use PaLM 540B (Chowdhery et al., 2022) in-context learning under the 5-shot setting with the prompt present below. The Entity_1 is the gold entity provided in the training set (with KB setting). The Entity_i i is the top-1 prediction from the entity linking stage of the queried image.
B.1 without-KB Models
PaLI and PaLI-X. We fine-tuned 17B PaLI (Chen et al., 2023b) and 55B PaLI-X (Chen al., 2023a) on INFOSEEK training set using the "answer in en: [question] <extra_id_0>" prompt.
Please answer the following question. question: (This is Entity_1. Question_1). answer: Answer_1].
BLIP2 and InstructBLIP. We fine-tuned a BLIP2 (Li et al., 2023b) and InstructBLIP (Dai et al., 2023) on INFOSEEK training set using the "Question: [question] Short answer:" prompt with the LAVIS library (Li et al., 2023a). The length penalty is set to -1. Since BLIP2 models present zero-shot capabilities on INFOSEEK, employ early stopping prevent over-fitting on the train- ing set based on the performance on the validation set.
question: (Thss is Entity_i. Question_i). answer:
FID. The T5 FID (Izacard and Grave, 2020)
model was fine-tuned in two stages using 100
passages with a maximum input length of 192 tokens. To form synthetic training data with (passage, question, answer) triples, we combine oracle entity passage with linked entity (from EntLinker) passages. We fine-tune the model on Natural Questions (Kwiatkowski et al., 2019) for 10k steps and then continue to fine-tune it on INFOSEEK for 200 steps with a batch size of 64. question: This is Entity. Question. context: Passage
OFA. We fine-tuned the OFAlage (Lu et al., 2022) model for 20k steps. During inference, we apply

---

C Additional Experiment Results
Complete numbers for With-KB Models. We show the complete results for With-KB models in Table 10.
Complete numbers for INFOSEEK Wikidata Val- idation set. We show the complete Validation re- sults for Without-KB and With-KB models in Ta- ble 11 and question type score of unseen entity split in Table 12.
C.1 OK-VQA Annotation Guidelines
Five adult annotators each annotate 100 examples (500 in total) sampled from the OK-VQA training set. Annotators are instructed to categorize each example into one three categories (see Table 13).

---

INFOSEEK Wiidd
INFOSEEK Ruman
Model
# Params
Components use
UNSEEN QUESTION
UNSBEN ENTITY
UNSBEN QUESTION
UNSEEN ENTITY
Overall
Overall
CLIP - PaLM
540B 1B
CLIP CLIP & FiD
21.9
18.6
20.1 19.3
15.6 18.9
14.9
15.2
CLIP >> Fi
20.7
18.1
17.6
18.2
Table 10: INFOSEEK full results With-KB setting.
INFOSEEK Wilkine
Model
UNSEEN
UNSEEN ENTITY
Overall
QUESTION
Without-KS Setting
Prior
4.6
3.2
PaLM (Q-only)
5.5
4.8
InstructBLIP
15.0
14.0
BLIP2
15.0
14.2
14.6
PaL.-17B
24.2
16.7
19.7
PaLI-X
25.8
22.4
24.0
With-KB Setting
CLIP PaLM
22.7
18.5
20.4
CLIP
23.3
20.9
Oracle FiD
52.1
53.0
52.5
Table II: INFOSEEK full results on Wikidata validation
set.
TIME (Acv.)
NUMERICAL
STRING
Model
(Reiaxed Avc.)
(Acc.)
No-KB Setting
Prior
0
3.5
2.3
PaLM (Q-only)
4.6
11.0
2.7
InstructBLIP
6.6
8.2
16.1
BLIP2
5.6
6.0
17.0
PaLI-17B
1.0
14.8
18.2
PaLI-X
8.1
17.2
24.8
With-KB Setting
CLIP - PaLM
17.8
21.3
17.7
CLIP + FiD
13.8
15.2
20.5
Table 12: Results w.r.t. each question types the INFOSEEKwatian val set of unseen entity split.
Question Category
Percentage
Answered directly by looking the correspandng image
50.8%
Answered without looking at the image only) Requiring Google search an answer
20%
29 29.2%
Table 13: OK-V --Q annotation results.


Reviews:
[{"review_id": "89esjCdki7", "reviewer": "Reviewer_Kp26", "summary": "", "questions": "1. when will the dataset be open source? It has been posted for almost half a year now？\n2.  Provide a detailed introduction to the model fine-tuning setting.\n3. The scale of the BLIP2 and PALI-X models and the difference in pre-training data are significant, the comparison between these two models unfair.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(Add. Experiment) Q1: Missing reference of \"The latest multimodal models such as MiniGPT4, LLaVA, mPLUG-Owl etc.\"**\n\nWe are grateful for Reviewer's expert insight into multimodal models and for highlighting seminal works like MiniGPT4, LLaVA, and mPLUG-Owl. We agree that these models have made significant contributions to the community. In response, we have expanded our evaluation to include a fine-tuned version of mPLUG-Owl on InfoSeek, and the results are included below (we will add them to camera-ready with details of the fine-tuning process).\n\nWe feature five multimodal models spanning publications from 2022 to 2023, the most recent of which was on arXiv a month before the EMNLP deadline. \nOur initial selection criteria were primarily guided by models with published SOTA results on VQAv2 or OK-VQA. However, we recognize that the models suggested by the reviewer have a particular emphasis on open-domain conversations, and we are in the process of including them in our evaluation for camera-ready.\n\n\n| Model | arXiv date | report results on VQAv2/OK-VQA? | InfoSeek_Human |\n|----------|---------------|:------------------:|-----------:|\n| OFA (L1192) | 2022/02/07 | Yes |  4.0 |\n| PaLI-17B | 2022/09/14|Yes| 8.2 |\n| BLIP2 | 2023/01/17 |Yes| 9.4|\n| LLaVA | 2023/04/17 |N\\A| todo |\n| MiniGPT4 | 2023/04/20 |N\\A| todo |\n| mPLUG-owl | 2023/04/27 |N\\A| 7.7 |\n| InstructBLIP|2023/05/11|Yes| 9.9 |\n| PaLI-X | 2023/05/29 |Yes| 10.8 |\n\n*- Experiment hypeparameters: LoRA (default) fine-tuning (MAGAer13/mplug-owl-llama-7b-pt) language decoder with lr=2e-4, batch size=8, 60k iterations, on a single A40 GPU.*\n\n---\n\n**(Clarification) Q2: \"when will the dataset be open source?\"**\n\n- We are fully committed to facilitating the availability of our dataset and would make the dataset **publicly available under the Apache-2.0 license** (upon the end of EMNLP anonymity period).\n\n---\n\n**(Clarification) Q3: \"The scale of the BLIP2 and PALI-X models and the difference in pre-training data are significant, the comparison between these two models unfair.\"**\n\nWe agree that this is a factor to be considered when reading the experiment, and would emphasize it in our experiment (though we already highlighted the model scale explicitly in Table 4). \n\nHowever, we want to emphasize that our mission is to help the community understand the current SoTA model’s capability in answering visual infoseek questions, so models are chosen based on their leading multimodal performances rather than pre-training recipes. \n\nMiniGPT4 and LLaVA (which were suggested above) are also not trained at a similar scale, but we agree it would be beneficial to include these experiments, with appropriate caveats highlighted the text and captions.\n\n**(Clarification) Q4: \"Provide a detailed introduction to the model fine-tuning setting.\"**\n\nDetailed instruction on the fine-tuning setting is in Appendix B.1 - \"Implementation details of the baseline - B.2 Without-KB models\" (L1178) and hyperparameters are summarized in Table 9.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper constructs over 1 million visual information-seeking QA pairs using the Wikidata database. Compared to previous works, this dataset covers more fine-grained knowledge and visual understanding, which could potentially accelerate community development.\n2. The paper validates the dataset's effectivenessthrough fine-tuning on VLPs.", "reasons_to_reject": "1. This work has limited innovation, except for proposing a dataset；\n2. This  evaluation has too few models，lacks some of the latest multimodal models including SFT such as MiniGPT4, LLaVA, etc.", "questions_to_authors": "1. when will the dataset be open source? It has been posted for almost half a year now？\n2.  Provide a detailed introduction to the model fine-tuning setting.\n3. The scale of the BLIP2 and PALI-X models and the difference in pre-training data are significant, the comparison between these two models unfair.", "paper_topic_and_main_contributions": "The paper proposes VQA dataset that focuses on visual info-seeking questions. The authors evaluate the zero-shot, fine-tuning, and retrieval performance of sota pre-trained models on the dataset.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "The latest multimodal models such as MiniGPT4, LLaVA, mPLUG-Owl etc."}, "extra_text": {}}, {"review_id": "WWB5J0GrOx", "reviewer": "Reviewer_fHav", "summary": "", "questions": "1) It seems odd that 70.8% OK-VQA questions can be answered by adults without search engines. How did you come to this conclusion? What were your experimental configurations? What kind of questions and images were involved? All these details are missing;\n\n2) During question writing, how many images are collected? How many questions are written for one image? It is claimed that 3-5 questions for a visual entity, yet an image involves multiple visual entities and how many questions are generated for one image is not clarified. Additionally, the entity-related Wiki articles have not been shown to the annotators during question generation, how to promise the answers to their questions can be extracted from these articles?\n\n3) INFOSEEK_Wikidata is claimed to be automatically generated, while its template generation and refining are both implemented by humans. Additionally, I’m not sure whether a dataset can be claimed as automatically generated if no language models are involved in generation.\n\n4) What are the potions of unseen entities and unseen questions in two data splits? A dataset is naturally challenging if part of its information is masked. It’s unclear the challenge of INFOSEEK_Wikidata comes from its intricate questions or its unseen information.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**\"Most efforts in building the new dataset are contributed by humans, which is consuming, non reproducible, and potentially non objective;\"**\n\nWe appreciate the reviewer's concern regarding the time and resource-intensive nature of human-annotated datasets. While it's true, it remains the *gold standard* in NLP and open-domain QA evaluations for ensuring high-quality data (e.g., TydiQA,  Natural Questions, SQuAD, OK-VQA all rely on human annotation).\nTo address the concern of cost, we have also developed an automatically constructed dataset, InfoSeek_Wikidata 1M.\n- **Reproducibility**: We commit to open-sourcing the dataset under the **Apache-2.0 license** upon the EMNLP anonymity period.\n- Quality: We conduct rigorous quality control during data collection with high quality evaluation accuracy at 95% (L259).\n\nBy balancing human-annotated and automatically constructed datasets, we aim to offer a comprehensive resource for the community.\n\n\n\n**(Clarification) Q1: I’m not sure whether a dataset can be claimed as automatically generated if no language models are involved in generation.**\n\nWe appreciate your observation regarding the term \"automatically generated\" in the context of our dataset creation. It's true that the recent surge in LLM-based data generation might set a particular expectation for the term. Our aim with this dataset is to maximize scalability while minimizing, but not entirely eliminating, human intervention for quality control. In light of your comment, we will revise the term to \"**automatically constructed**\" in the camera-ready version to clarify our approach. Thank you for bringing this to our attention.\n\n**(Clarification) Q2: Details of experiment setting of \"70.8% OK-VQA questions can be answered by adults without search engines\"**\n\nWe will include comprehensive details in the appendix for further clarification. Additionally, we invite the reviewer to check some examples from the OK-VQA data browser via the provided link (https://okvqa.allenai.org/browse.html - e.g., \"Question: Did he just block or catch the ball?\").\n\nStudy details:\n- Five adult annotators each annotate 100 examples\n- In total, 500 randomly sampled examples from the OK-VQA training set\n- Annotators are instructed to categorize each example into one of three categories: \n|Category| Percentage|\n|------------|-----------|\n|Questions that can be answered directly by looking at the corresponding image. | 50.8%|\n|Questions that can be answered without looking at the image. | 20%|\n|Questions requiring a Google search for an answer.| 29.2%|\n||\n\n\n**(Clarification) Q3: What are the potions of unseen entities and unseen questions in two data splits?**\n\nPlease find stats in Table 7:\n|Split| Unseen Question | Unseen Entity|\n|-----|-------------------------|--------------------|\n|Val | 18,656 | 54,964 |\n|Test | 98,901 | 249,079| \n|Human | 3,248 | 5,683|\n||\n\nThe design principle of InfoSeek is to evaluate \"pre-training knowledge\" in the multimodal models (L288-290), which is intentionally aligned with the goal of assessing world knowledge in LLMs through open-domain text QA (e.g., TriviaQA/Natural Question in LLaMA-2 report). In this context, the SOTA models achieving top performance on TriviaQA are LLMs that use few-shot prompting, rather than models that are fine-tuned on training set (https://paperswithcode.com/sota/question-answering-on-triviaqa). Similarly in InfoSeek, the unseen questions/entities serve to rigorously assess the generalization abilities of the multimodal models, preventing model overfitting to the training set.\n\n**(Clarification) Q4: how to promise the answers to their questions can be extracted from these articles? and how many questions are generated for one image?**\n\nWe ensure that only questions answerable by the articles are included in our dataset, following the TyDiQA-GoldP annotation guidelines (L 1029-1033).\nThe main reason we don't show annotators Wiki articles during the question writing stage is to ensure they have no prior knowledge about the question they ask, thus maintaining the info-seeking nature of the question.\n\nBelow, we report additional stats and plan to add in the camera-ready version:\n| Split | Question/Image|\n|-----|-----------------|\n|InfoSeek Wikidata| 1.4|\n|InfoSeek Human|1.0|\n||\n\n*Reference*\n\n- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension, Joshi et al., 2017 ACL\n- Natural Questions: A Benchmark for Question Answering Research, Kwiatkowski et al., 2019, TACL\n- SQuAD: 100,000+ Questions for Machine Comprehension of Text, Rajpurkar et al., 2016, EMNLP\n- TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages, Clark et al., 2020, TACL", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The paper is mostly well organized and easy to follow;\n\n2) The novel dataset may illuminate the VQA research with cross-modal knowledge;\n\n3) The experimental evaluations are relatively satisfied.", "reasons_to_reject": "1) Most efforts in building the new dataset are contributed by humans, which is consuming, nonreproducible, and potentially nonobjective;\n\n2) Some descriptions seem obscure or not rigorous;\n\n3) The soundness of the new dataset needs a clarification.", "questions_to_authors": "1) It seems odd that 70.8% OK-VQA questions can be answered by adults without search engines. How did you come to this conclusion? What were your experimental configurations? What kind of questions and images were involved? All these details are missing;\n\n2) During question writing, how many images are collected? How many questions are written for one image? It is claimed that 3-5 questions for a visual entity, yet an image involves multiple visual entities and how many questions are generated for one image is not clarified. Additionally, the entity-related Wiki articles have not been shown to the annotators during question generation, how to promise the answers to their questions can be extracted from these articles?\n\n3) INFOSEEK_Wikidata is claimed to be automatically generated, while its template generation and refining are both implemented by humans. Additionally, I’m not sure whether a dataset can be claimed as automatically generated if no language models are involved in generation.\n\n4) What are the potions of unseen entities and unseen questions in two data splits? A dataset is naturally challenging if part of its information is masked. It’s unclear the challenge of INFOSEEK_Wikidata comes from its intricate questions or its unseen information.", "paper_topic_and_main_contributions": "A visual question answering dataset is introduced in this paper for visual information-seeking questions. Various pre-trained multi-modal models are evaluated on this dataset and fine-tuned by using fine-grained knowledge, validating the challenge of the dataset in posing intricate questions.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UM7lXQYLP5", "reviewer": "Reviewer_Vbdw", "summary": "", "questions": "1. Your dataset is open-ended or multiple-choice? why the random model can achieve 0.1 or 0.2 accuracy?\n2. Can you provide some methods to fix information-seeking questions (as for the sota model just achieve almost 20% accuracy)?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**(Clarification) Q1 : \"Your dataset is open-ended or multiple-choice? why the random model can achieve 0.1 or 0.2 accuracy?\"**\n\nOur dataset consists of open-ended questions that require short text answer spans. Further illustrations and examples can be found in Figure 7 of the manuscript. As for the random baseline achieving 0.1 to 0.2 accuracy, it is a result of sampling answers from the training set. Due to the nature of the data, some answers happen to be correct purely by chance (e.g., common country name).\n\n**Q2: \"Can you provide some methods to fix information-seeking questions (as for the sota model just achieve almost 20% accuracy)?\"**\n\n- **InfoSeek**: The primary focus of our paper is to identify and quantify the performance gap in current SOTA models on visual information-seeking questions. Our goal is to design the task and introduce a dataset for future research and outlining the limitations of existing models.\n- **End-to-end & Retrieval Model**: We believe that emerging pre-trained multimodal models equipped with more efficient pre-training computation (without-KB setting) or retrieval-augmented methods (with-KB setting) will show significant improvements in accuracy.\n- **Retrieval methods we tried**: To offer additional context on the room for improvement, we have estimated an upper bound of over 50% accuracy with a retrieval-based models (Table 5). This suggests an improved CLIP-based image retrieval model could lead to substantial improvement on InfoSeek.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper presents a new vqa dataset.\n2. The experiments are abundant.", "reasons_to_reject": "Maybe you can provide a method to improve the performance on your dataset.", "questions_to_authors": "1. Your dataset is open-ended or multiple-choice? why the random model can achieve 0.1 or 0.2 accuracy?\n2. Can you provide some methods to fix information-seeking questions (as for the sota model just achieve almost 20% accuracy)?", "paper_topic_and_main_contributions": "This paper presents a VQA dataset INFOSEEK, that focuses on visual info-seeking questions. Besides, this paper analyze the ability of state\nof-the-art models to answer visual info-seeking questions.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a dataset on challenging the open-knowledge of VL models. The purpose of this dataset is clear to me, and I like the OKVQA comparisons. As a dataset paper, the paper's method part also nicely elucidates the status of existing approaches, which compares the zero-shot, fine-tuning (i.e., like using model's internal KB), and explicit KB. The overall discussion went positive on the decision of this paper.
Decision: Accept-Main