MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering
Authors: MAHMOUD KHADEMI
Ziyi Yang
Felipe Vieira Frujeri
Chenguang Zhu
Venue: EMNLP 2023 Findings (2023)
Keywords: Vision-Language Models
Large Language Models
Visual Question Answering

Abstract: Thanks to the strong reasoning capabilities of Large Language Models (LLMs), recent approaches to knowledge-based visual question answering (KVQA) utilize LLMs with a global caption of an input image to answer a question. However, these approaches may miss key visual information that is not captured by the caption. Moreover, they cannot fully utilize the visual information required to answer the question. To address these issues, we introduce a new framework called Multi-Modal Knowledge-Aware Reasoner (MM-Reasoner) for KVQA. MM-Reasoner first utilizes a set of vision APIs, such as dense captioners, object detectors, and OCR, to extract detailed information from the image in textual format. Then, it prompts an LLM to extract query-specific knowledge from the extracted textual information to provide a rich representation that contains external knowledge, commonsense, explicit supporting facts, and rationales required for reasoning. Finally, the knowledge, query, and visual input are used to fine-tune a Vision-Language Model (VLM). At test time, MM-Reasoner uses the potential answers predicted by the VLM to iteratively update and optimize the prompt, refining its answer. Empirical studies show that MM-Reasoner achieves state-of-the-art performance on several KVQA datasets.

MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Basee Visual Question Answering
Mahmoud Khademi, Ziyi Yang, Felipe Vieira Frujeri, Chenguang Zhu
Microsoft Cognitive Services Research Group
(mkhademi, ziyiyang, fevieira, chezhu)emicrosofft com
Abstract
Thanks to the strong reasoning capabilities of Large Language Models (LLMs), recent ap- proaches to knowledge based visual question answering (KVQA) utilize LLMS with global caption of input image answer ques- tion. However, these approaches may miss key visual information that not captured by the caption. Moreover, they cannot fully uti- lize the visual information required to answer the question. To address these issues, in- troduce new framework called Multi-Modal Knowledge-Awer Aware Reasoner (MM-R Reasoner) for KVQA. MM-Reasoner first utilizes of vision APIs, such as dense captioners, object detectors, and OCR, extract detailed infor- mation from the image in textual format. Then, it prompts an L.M to extract query-specific knowledge from the extracted textual informa- tion to provide a rich representation that con- tains external knowledge, commonsense, ex- plicit supporting facts, and rationales required for reasoning. Finally, the knowledge, query, and visual imput are used to fine-tune Vision- Language Model (VLM). At test time, MM- Reasoner uses the potential answers predicted by the VLM to iteratively update and optimize the prompt, refining its answer. Empirical stud- ies show that MM-Reasoner achieves state-of- the-art performance on several KVQA datasets.
Vision APIs: Deuse Captions, OCR, Objects, Tags, People
('CENSE CAPTIWE"" "l''ottont'' "a pexcon on movuooard coing down alope" 1, "contidenae's 0,54, 'boarding box"s [8) 640, 48C]),
('content person okiing down clup', "conErSenze's 0.40, 'bounding box't 1261, 77, 107, 24911], tock"; [].
"OOMECI"": Li'mame's "person', 'contidence's 0.30, "hoarding box":
84, 123, 18011, ('aswe's "smoddoesd", "aumEidwmuw'' 0.97, "bounding hos's [211, 201, 185, #2]
"Tare" [1"ieg name"; "'amm"", "manf fileeres 0.99), g"teq name" 'spart". "conEidence"s 0, 89),
"tag name" 'wintor', "ocotidoz.e" 0.921],
"PSOFiE"s /l"coniidenee'' 3.88, 'bounding boot":
[248, 74, 124, 1551), (*ooaridanca": 0.72, "banndin mg box! [624, 27, 15, 28]]]]
G: Why 1s his whole body covered?
Prompt: Instructions and In-confext Examples
Large Language (CPT 4(Vicuna)
["hationalion aosts "rim person in somndbsse my and noeds to stay warm and protected from cold possizile tolls. wearing clothing
cotrecs their cntire
boday. The enesboarder in trying to protect the r body from sold and posa ibin injuriea the slope,
Vision Language Model (Flamingaat Code)
1 Introduction
("eorstiLel ANSMERSs: ; [i ppooeectionas 0.501,
The knowledge-based visual question answering (KVQA) task (Marino et al., 2019) requires not only visual information from the image, such as object attributes and visual relationship informa- tion, but also external knowledge, commonsense, rationales, and supporting facts for reasoning and predicting the correct answer. Therefore, typ- ical KVQA model consists of knowledge re- trieval module and an answer module. The tradi- tional knowledge retrieval module usually retrieves knowledge from sources such as Wikipedia, knowl- edge graphs, and web search (Wu et al., 2022).
I'stay warm!: 0.251, ("cold's 0.20313
Figure 1: Multi-Modal Knowledge-Aware Reasoner
More recently, Large Language Models (LLMs) such as GPT-3 are used to produce related knowl- edge (Lin et al., 2022; Hu al., 2022b). The latter approach is preferred since traditional knowledge retrieval often introduces irrelevant information to

---

the question (Yang et al., 2022a). After obtain- ing the required knowledge, the answer module takes the visual input or global caption of the image, query, and knowledge to predict an answer. The answer module either a transformer encoder- decoder architecture with trainable parameters (Lin et al., 2022) or more recently, frozen LLM such GPT-3 (Yang et al., 2022a; Hu et al., 2022a).
STOP HAMMERTINE
However, there are several key challenges to the above approaches. First, when using an L.LM, the input needs to be text, but the recent upproach of feeding the global caption of the input image into LLMs (Hu et al., 2022a) may miss key visual infor- mation. Second, the conceptual diversity of visual questions often necessitates multimodal reasoning with the integration of various external information beyond the image itself. These diverse types of knowledge include: commonsense, knowledge of concepts represented visually, external Knowledge, e.g., Marco van Basten is former football player born in the Netherlands., insights about the world's physics, e.g., Mountain areas have a lower temperature than other areas, etc. Third, recent approaches cannot fully utilize visual information in their answer module since they use an LLM to predict an answer. These ap- proaches often struggle to answer visual questions that require fine- gra-ggaine visual information, e.g., spatial relationships and object attributes, or knowl- edge of concepts represented visually, c.g., What is the pattern of the table cloth called?
'00R1: [( [("content': '870P", "contidance's
0.391, ('ecatent's "HABERTIN"" 'cnnEicaace's 3.6711
in the picture? Ratioraless The word Ramertine is wrieten cn the stop sign. xP Hermer saag the 2cnz V Can't 2ouch This which Bamert
eang neag
GT: ne Hamner MM-R-1 unknown, MM-Reasoners
"OSuSEALTY RE006N13MR'S s [("oame' 'Bazoo van
Basten", "bounding boxt [241, $3, 126, 2/112, (teame"s 'Klaas-Jan Ranteisar', "bounding box!: (1212, 235, 112, 161]]]
Q: Berw wil the piogle he in the lnage born in the swmes country?
sotellee The people in inage Narco
Bexten and itlaws san Blutelaan,, who vere both bore: in the Wetherlands. Tteretore, they vere barn in the same country.
yes, 100-B-1 unknown, So-Res yes
Figure 2: OCR and celebrity recognizer are helpful for answering questions. GT refers to the ground truth, and MM-R is baseline model similar AMM-easoner, but it uses global caption instead of vision APIs.
To answer a diverse set of questions, a KVQA model requires (1) vision expertise, e.g., celebrity recognition, optical character recognition (OCR), and object detection (2) the reasoning capacity and world knowledge of the recent LLMs, and (3) the ability of Vision-Language Models (VLMs) to generate rich joint representations of vision and language. However, current knowledge-basee VQA models cannot provide all these capabili- ties within a unified framework. To address these challenges, we introduce new framework, called Multi-Modal Knowledge-Aware Reasoner (MM- Reasoner), for knowledge- VQA (Figure 1).
extract query-specific knowledge from the textual representations in a supervupe way via prompt designing, which requires only a few in-context ex- amples. The LLM can provide rich representation that contains external knowledge, commonsense, explicit supporting facts, and rationales required for reasoning. Finally, the extracted knowledge, the question, and the regional visual information of the input image are used to fine-tune a VLM. The VLM can fully utilize fine-grained visual in- formation such as object regions, visual relation- ships/attributes, and visual spatial information sup- plementary to the textual representation to answer the question (see Figure 3). At test time, MM- Reasoner uses candidate answers predicted by the VLM iteratively update and optimize the prompt to refine its final answer. This allows our frame- work to effectively exchange knowledge and visual information between the LLM and the VLM.
The MM-Reasoner first leverages a rich set of vi- sion APis/expertise such as dense captioner, object detector, tag detector, OCR, people detector and celebrity recognizer, obtain detailed textual rep- resentation of the visual input required produce more accurate answers (Figure 2).
Then, the MM-Reasoner leverages an LLM to

---

(1) We introduce new framework for KVQA that offers vision APIs, the reasoning power of LLMs, and the rich joint representation of VLMs, all in a unified and composable framework.
(2) A key novelty of MM-Reasoner is that it effectively utilizes visual features in generating knowledge and predicting the final answer by lever- aging a VLM and exchanging visual information and knowledge between the VLM and the LLM. (3)
"DEESE CAPTIONB's content'' "a persen drinking fron
a class table with food', "cortidence"; 0,32, "bounding box'; [o, o, 6121), s..
The MM-Reasoner achieves statc-of-the-art performance on several knowledge-basel VQA benchmarks.
9: What the pattert of the sable cloth called? Ratioasies: Caa't detezmine the pattezn et the table clath from the inage.
er: checkeered LIX:: can't deternine, W Beasoner::: abackered
2 Related Work
Our work is related to several recent studies in the area of multi-modal learning and LLMs.
Knowledge-Based VQA. In REVIVE (Lin et al., 2022), the authors proposed to first employ an ob- ject detector to locate the objects, and then use the cropped bounding-bov proposals to retrieve various types of external knowledge. Finally, they merged this knowledge with the regional visual features into transformer to predict answer. In Hu et al. (2022a), the authors introduced a captioning model called PromptCap, which is designed to generate question-aware captions using a GPT-3 prompt and question-answa pairs from existing VQA datasets. However, to generate the training examples, the PromptCap still relies on global captions of the image. Additionally, the visual features are not effectively utilized in the final answering model, as the answer module is frozen LLM. In con- trast, our prompt design enables our framework to extract query-specific knowledge from the visual input, avoiding the extraction of unrelated infor- mation. Additionally, MM-Reasoner harnesses the capabilities of VLMs such as Flamingo (Alayrac et al., 2022) and i-Code (Yang et al., 2022b) to incorporate visual features in the answer module. In Hu al. (2022b), the authors proposed an end-to-end model, called Retrieval-Augmented Vi- sual Language Model (REVEAL), that learns to encode multimodal world knowledge into a mem- ory and utilize it for addressing visual questions that demand extensive knowledge. REVEAL is composed of four modules: the memory, the en- coder, the retriever, and the generator. The memory encodes different types of multimodal knowledge via a shared encoder. The retriever finds the most relevant information, and the generator uses that information with the input question to predict the
"DENGE CAPTIONS'; Ll'ccatsatt: child catching tootball ball", 'contidence's 0,502, "boundiag_boos" box"" D3, 646, 6251 sear
9: what position does he play?
katicnales; The ctild 15 catching cortball ball. Bosition
specified,
GT: goalte, 1MM+: unkraun, MM-Roasoner: goa
MM-Reasoner can effectively utilize visual features to find the correct answer by leveraging VLM as its answer module. GT is ground truth and LLM+ is a baseline model that prompts GPT-4 with textual descriptions obtained from vision APIs and the question to generate an answer. LLM+ fails since textual de- scriptions from APIs do not provide useful information.
Figure 3:
MM-Reasoner is an integrative and compos- able framework that works with variety of ex- isting LLMs and VLMs. Since these LLMs and VLMs have already been trained on large-scale datasets, the fine-tuning cost of MM-Reasr is much lower than that of existing models. We have conducted extensive empirical studies on MM-Reasoner using several datasets. The results show MM-Reasoner achieves state-of-the-an art ac- curacy on the KVQA task, achieving 60.8% ac- curacy the OK-VQA dataset, 60.2% on the A- OKVQA dataset, and 40.0% in the few-shot set- ting on the OK-VQA dataset. The project page is available at https://gi thub. com/microsoff i teade//ttee/mai//mmeeee the key contributions of this paper are as follows:
In summary,

---

final answer. The model is pre-trained on a large amount of data and then fine-tuned for specific tasks. However, REVEAL does not leverage the reasoning capabilities of recent LLMs.
MM-REACT can effectively solve the given rea- soning tasks, especially when tailored to the vision APIs. For instance, by employing an OCR API, can gather information from multiple receipts and calculate the total cost. However, since MM- REACT relies solely on textual representation of the input image and uses frozen LLM to generate an answer, it often struggles to answer questions that require fine-grained visual information.
Multimodal Composable Frameworks. Re- cently, the composition of large pre-trained models has been extensively studicd. The prevalent ap- proach for composing these models is through joint fine-tuning on new tasks. Yang al. (2022h) intro- duced i-Code, a self-supervised pretraining frame- work that allows users to merge vision, speech, and language modalities into comprehensive and ver- satile vector representations. In the i-Code frame- work, data from each modality is initially fed into pretrained Lingeemodalle encoders. The outputs from these encoders are subsequently combined using a multimodal fusion network that employs attention mechanisms and other novel architectural designs to efficiently integrate information across modalities. The i-Code is pretrained end-to-end us- ing objectives such as masked modality unit mod- eling and cross- erosatmy contrastive learning.
Multi-Modal Knowledge-Aware Reasoner In this section, we present the Multi-Modal Knowledge-Aware Reasoner framework. As Fig- ure shows, MM-Reasoner is composed of three components: set of vision APIs, an LLM, and a VLM. It is worth mentioning that is an integrative, modular, and composable frame- work that works with variety of existing vision APIs, LLMs and VLMs. These components work together to extract intrinsic information from the image and query, extrinsic information from exter- nal knowledge, and conduct visual-textual reason- ing to produce the answer. In the following, we explain these components in more details.
Wang al. (2021) introduced VLM frame- work, called SiMVLM that is trained end-to-end using a single language modeling objective, simpli- fying the training process with large-scale weak su- pervision. Alayrac et al. (2022) proposed Flamingo, a set of VLMs capable of connecting pre-trained vision-only and language-only models, processing sequences of interleaved visual and textual data, and accepting images videos as inputs. How- ever, these methods can be computationally de- manding. In contrast, models can be composed through a shared modality such as language. Zeng et al. (2022) proposed Socratic Models, a modular framework that allows multiple pre-trained models to exchange information, acquire new multimodal capabilities without fine-tuning.
3.1 Vision APIs
The goal of vision APIs is to extract detailed visual information from the image in textual format, to be later used in LLM prompts. In our experiments, MM- MMM-Rcasoner adopts Azure Cognitive Services for Vision (ACS Vision) The ACS Vision is unified service that offers innovative computer vision capa- bilities. Our framework utilizes the following APIs from the Image Analysis SDK:
Dense captions Generates aggion-grounded captions for all important objects detected in an image.
LLMs with Plugins and Tools. Our frame- work is related the use of tools and plugins for LLMs. Toolformer (Schick et al., 2023), the au- thors demonstrated that language models can self- learn the utilization of external tools through sim- ple APIs, such as calculators, search engines, and translators. Toolformer is designed to determine the appropriate APIs to invoke and the specific ar- guments provide via a few in-context examples. In MM-REACT (Yang et al., 2023a), the authors in- tegrated specialized vision APIs with ChatGPT to solve various visual understanding tasks. Through several demonstration examples, they showed that
Tags Extracts common tags from an image based on thousands of recognizable objects, living beings, scenery, and actions.
Objects Object detection is similar to tag- ging, but the API returns the bounding box coordinates for each tag applied.
OCR Extracts printed and handwritten text from an image.
'nttps://azure.tt www.rreeeeiioooooo wtwp://tt..commeeeess////rueeeee

---

People- Extracts bounding box coordinates of each detected person in an image.
their confidence scores. For closed-vocabulary VLM, this can be obtained by choosing small subset of answers with the highest logits. For an open- vocabuloyy VLM, this list can be obtained using a beam search decoder or by fine-tuning mul- tiple instances of the VLM and utilizing greedy search decoder for each instance of the VLM. We use the latter method in our experiments. More specifically, we prompt the LLM times to get possibly different sets of rationales and fine-tune m different instances of the VLM.
Celebrity Recognizer Detects and identifies known celebrities in an image.
We aggregate the results from the Vision APIs to summarize the image with a set of captions, a set of objects/persons present in the image, a set of tags associated with the image, set of texts written in the image, and the location of the detected people.
3.2 Large Language Model
3,4 Further Improvements
Given the vision API outputs, we design a vision- informed language prompt to elicit rationales from LLM helpful to answer the question. The prompt is composed of instructions, a few in-context ex- ample instances, and the current input instance. Each instance includes an example question and the outputs of the vision APIs for the correspond- ing example image. Figure 4 shows prompt that we use for the OK-VQA dataset. More specifically, given a question and visual input pair denoted by (V; Q), the input to the LLM is prompt denoted by obtained as follows:
Iterative Reasoning. At test time, we first use the above process to obtain set of answers along with their confidence scores. The confidence score for each answer is acquired using the probability distri- bution over the predicted answers by the instances of the VLM. Instead of directly returning the an- swer with the highest confidence score as the final answer, we discovered that performance be sig- nificantly improved by updating the prompt with the first pass prediction and repeating this process. Specifically, we initially add answers with their confidence scores to the prompt and feed the up- dated prompt into the LLM. Then, we obtain a new of rationales. These are often more informa- tive than the previous rationales since the updated prompt provides VLM's answer suggestions. Next, we feed the new rationales into the instances of the VLM to acquire an updated set of potential answers. After repeating this process for a few iterations, we use the answer with the highest score the fi- nal answer. We use two reasoning iterations after obtaining the initial answer. In our experiments. applying more iterations did not help improve the performance. During the iterative reasoning, the instances of the VLM are kept frozen.
concat (I: (Xici Yoo)f:
(1)
where, I denotes the instructions, (Xici Y Yic) list of k in-context input-output pairs, and
X = concat (APIs(V);
(2)
is obtained by appending the question Q to the outputs of the vision APIs for visual input V. In- serting & in-context onput- pairs before the X substantially improves the performance. With the prompt, the LLM utilize its strong reasoning ca- pacity to respond with rationales. These rationales be commonsense knowledge, external informa- tion, basic factual knowledge, supporting facts, or any information from the image that is required or helpful to answer the question.
k-NN-Based In-context Examples. We apply dynamic set of in-context examples during test time. First, we use a fine-tuned VLM to extract an embedding for each image and question pair in the dataset, utilizing the average encoder embedding. Then, for each test example, we select k training examples that are closest to the test example based the Euclidean distance between the example embeddings. These training examples are, in turn, used as in-context examples for this test example. The intuition to use better in-context demonstra- tions by retrieving examples that are semantically similar to the test input (Liu et al., 2021).
3.3 Vision Language Model
Next, we feed the rationales, the image, and the question into the VLM. The VLM is fine-tuned on the training data to adapt to the rich input format. In this way, both the original visual information and external knowledge are leveraged produce the answer. The framework supports both open- vocabulary and closed-vocabulary VLMs. The VLM's output is a list of potential answers with

---

The following a conversation with an Al asaistant, Given an image description and question about the image, the assistant provides rationales that are required or helpful to answer the queation correctly, There raticmlcc can be commonsence knowledge, external information, basic factusl knowledge, supporting facts, or any informaiio from the image that is required helpful ta answer the question, Rach image is describer bed with set of captions, present in the image, a set of Laga associnted with the image, image, and the locatior of the people detected in the image, An empty 11st 1a represented by D], The Location of each ceplion, cbject, und delected pezson is apecified by bounding bex. bounding box 18 rectangle that surrounds an object, caption, or person in an image. Each bounding box is represented by a List of 4 numbero as 1x, y, w, hJ, where (8, y) corresponds to the x and y coordinates of the top-left corner of the rectangle, w represents the width of the bounding box, and h represents the height of the bounding box. the y, and h are nomalized to be between 0 and 1. 80, (0, 0J and (I, 1) corcespond to the and y coordinates of the top- left corner and botton-right cerner of the image, respestively. Each bourding box has a con fieeed seare that, indicatas how likely the objest, caption, or persor actually ly present that bounding box, In each Lurn of the conversalivn, the human a question about the image via a dictienary in the Python language, The question has a unique id, The NI reeponds with a dictionary in the Python language, The diclionsry has 3 keys: "question id", "rationales", and "potential answers". it an image description lacks sutticient intormation, the AI must still predict a 1ist of potential answers to the question and provide rationalcs.
net of objerts set of texts writlen in the
deseribes inage and seks
Human: The description of the image and the given question are: ("captiors"; [["caption": "a snowboarder going dowa a slope", "cosfideace"; 0.5, "bounding box"; 10.0, 0.0, 1.0, 1.011, ("capi.ion": "a snowboarder in the anow", "conf "ponfideecs"" D.5, "boundi ng box": [0.2, 0.1, 0.3, 0.411, ("caplion":
close-up of a person's a Lool", "comiidence"" 0.5, "bouading box"; 10.2, 0.3, 0,1, 0,2111, "cojecta"; [["obfect"; "pecaon", "contidence": 0.8, "bounsing box": 10 0.1, 0.3, 0,41H1, "tags": [["Lag"; "snow", "confidence"; 1.0), ("tag"; "sport", "confidence"; 1.01, ("tag": "outdoor", "conridence"; 1,01, I"taq": "person", "confidence": 1.0), ("taq": "extreme sport", "contidence"; 0.9), ("tag": "snowboard", "contidence"; 0.9), ("tag": "riding", "contidence": 0.9), ("tag": "piste", "contidence"; 0.9), ("taq": "winter sport", "contidence"; "vinter", 0,9), l"tag": "adventure", "confidence"; 0,91, "" "confidence"; 0.91, ("tag": "#l opestyle", "canf iderent; 0.9), ("sag": "fontwear", "conf Idencede f..9,, ("tag": "glacial landfurm", "confidence"; 0.9], ("Lag": "recreation Lion", "confidence": 0.0), ("Lag": "ski". "confi dennee: 0.8), ("Lag": "frenzing", "conf iddence: 0.8), ("Lag": "ouldoor recreation", "con "conIideeze" 0.B), ("Lag": "slope", "confidence"; 0.8), ("Lay": "mountsin", "confideuce"; 0.6), ("Lag"; "saowboarding", "confidence"; 0.6), I"Lay"; "skiing", "conf idenve"; 0.511, "texts"; U], "people Location": """onniddence: 0.9, "bounding box"; 10.2, 0.1, 0.3, 0.41H "question"; "Which season do you tind this weather?", "question id"; 42395], Please provide the rationalcs and a 11st of potential answers for the queation.
AT: ("question id"; 42345, "ratinnalels", "There in snow in the image. Weather in anowy typically in Winter.", "potential answrrs"; ["Winter"])
Humon: The description of the image and the givon qucation are: ("caption"; "horaco running on
hill", "coafidence" : 0,4, eq box": 10.0, 0.0, 1.9, 1.01;, ("caption"; "a horse zunning in
field", "confidenos"; 0.4, "bonnding ng box": [0.2, 0.4, 0.2, 0_2]], ( ("eaption":
horse with
large belly", *ean iidereed 0.3, "bounding box": 10.5, 0.4, 0.2, 0.211, ["capLion": "a horse
with tail" "confiderce"s 0.3, "bounding box": [0.4, 0.4, 0.1, 0.11), ("caption": "a horse
running in the grass", "conf idence": 0.4, "bounding box": 11.8, 0.2, 0.2, 0.11l1. "objecls";
[["objest"; "mammal" "confidence"" 0.0, "bounding box": [0.6, 0.a, 0.2, 0.11h, ("object":
"nammsi", "confidence". 0.8, "bounding box": 10.2, 0.4, 0.2, 0.113, ("object": "nanmai", "contidenoe"; ; 0.6, "bounding box"; 10.4, 0.4, 0.1, 0.1]J, ["obicct": "horge", "contidenoo"s 0.6, "bounding box": 10.5, C.4, 0.2, 0.2] "taqs": li"taq": "cataoor", "contidence": 1.0), ("tad": "grasa", "contidonce": 1.0), ("tag": "mamaa"" "confidenoo"" 1.01, ["tag": "animal", "contidence": 1.0), ("tad": "Livestock", "confidence": 0.9), ("tad": "Eield", "confidence": 0.9), ), I"tag": "hard", "confidenan"" 0.9), ("tag": "ranch", "confidence": 0.9), ("tag": "pasture", Pronfidence" 0.9), ("Lag": "sly", "confidenan" 0.9), ("Lag:: "herding". "confi dence": 0.9), ("tag": "hill", , "confidence"; 0.8), ("tag": "mountair", "confidence"; 0.8), ("Lag": "landscape", "confidence"" 0.7), ("Lag": "grazing", "confidence": 0.7), ("Lag": "hillside", "confidence"; 0.71, ("tag": "horse", "confidence"; 0.6), ["eag": "granny", "contidence": 0.67 I. "texts": II, "people location": II, "question": "What animals are these?", "question id": buggxttgtasppoc//cccoo "J, Please provide the rationales and 11st of potential answers for the question.
The prompt used in MM-Reasoner for OK-VQA dataset. The black text provides instructions, while the blue and red texts demonstrate in in-context examples and the current training example, respectively. At test time, we provide the potential answers predicted by the VML in addition to the image descriptions and question. The potential answers predicted by the LLM are not fed into the VLM. They only used for evaluating baseline.
Figure 4:

---

4 Experiments
we performed an ablation study by implementing five baseline models: LLM", VLM*, MM-R MM-R*, and MM-R (see Table 1).
This section covers the explanation of datasets and baseline models, followed by discussion of exper- imental results and implementation details.
In LLM ', the VLM component is removed. Hence, LLM+ does use the image directly generate an answer. This baseline is designed study the significance of the VLM module in our framework. prompt is constructed using the vi- sion APIs and the given question. Each in-context example is modified so that GPT-4 provides a sin- gle answer to the given question in addition to the rationales. For each example in the test set, the prompt is fed into the GPT-4. Then, the predicted answer is mapped one of the words in the test set vocabulary based on the cosine similarity be- tween the embedding the predicted answer and the embedding of each word in the vocabulary. We use this method because the LLM's answers often differ from the ground truth, even when an answer is semantically correct. We use GloVe (Pennington et al., 2014) to obtain the embedding for the pre- dicted answer and each word in the vocabulary. No fine-tuning is applied to LLM
4.1 Datasets
We evaluate MM-Reasoner's performance on the following datasets: OK-VQA (Marino et al., 2019), A-OKVQA (Schwenk al., 2022), FVQA (Wang al., 2017), and KVQA (Shah et al., 2019). OK- VQA a dataset for visual question answering that needs external knowledge to answer questions. OK-VQA includes 9,009 training questions and 5,046 test questions. A-OKVQA dataset the aug- mented successor of OK -K-V A-OKVQA ques- tions are challenging, encompass wide range of concepts, and unlike other knowledge- visual question answering datasets, they cannot be addressed by merely consulting a knowledge base The A-OKVQA offers rationales for answers in the training set. The rationales provide facts and snippets of reasoning required to answer the question. The train/validation/tess split of the A- OKVQA dataset includes 17,056/1, 145/,,, 702 ques- tions. There is overlap between question-image pairs in A-OKVQA and OK-VQA.
Baseline
APIs
LLM
VLM
IT-R
Acc
LLM
V
VA
X
X
38.8
FVQA is a VQA dataset that mostly contains questions requiring external knowledge to answer, and provides supporting fact triplets alongside the seree--uusttoon-nswwee triplets. Following (Wang et al., 2017), we used 1,090 test images, amount- ing to 2,899 questions. The KVQA (Shah et al., 2019) dataset contains 183,007 pairs of questions and answers, which involve 18,880 named entities and 24,602 images. The questions in this dataset require complex reasoning over large knowledge graphs, involving multiple entities, relations, and hops to arrive at an answer. KVQA is currently one of the largest datasets available for studying VQA over knowledge graphs. KVQA has 5 different train/validationn/ess splits. We use the last one.
VLM+
X
V
x
53.8
AVA
MM-R
57.1
MM-R
V
58.5
MM-R
V
V
59.2
Table 1: Ablation study on OK-VQA dataset. IT-R denotes iterative reasoning.
In VLM+, the LLM module is removed. The input to the i-Code v2 is the output of the vision APIs, the image, and the given question. VLM does not exploit the reasoning capabilities of the LLM module, and its external knowledge is limited to the VLM pre-trained knowledge only. VLM is fine-tuned with the ground truth answers. MM- R is a single MM-Reasoner framework without leveraging ensemble learning. MM-R* is similar to the MM-R, except that at test time, the output of the VLM is used as the final answer. That is, the VLM does not update the initial prompt. This baseline is designed to study the significance of the iterative reasoning in our framework. Finally, the MM-R- is similar to MM-R*, except that we only use a global image caption and the question to construct the prompt. This baseline is designed to study the
4.2 Baselines
We compared our framework with several models that have recently been developed, including the statc-of-the-art models PrompiCap, Flamingo, and REVIVE. For further comparison, we also include models that were proposed more recently in Arxiv preprints during the preparation of this work: Prophet and REVEAL. Additionally, to verify the significance of each component in our framework,

---

the question requires fine-grained visual informa- tion not described by the APIs. However, MM- Reasoner can find the answer by leveraging the power of its VLM. Figure shows our framework can refine its answer via the iterative reasoning.
The results of our ablation study are summarized in Table 1. LLM* performs poorly because the tex- tual representation of the image alone does not provide sufficient fine-grained visual information for reasoning. MM-R outperforms both LLM and VLM*, as can leverage the power both LLMs and VLMs. Additionally, MM-R outper- forms MM-R demonstrating that vision APIs can help improve performance. Lastly, MM-R surpasses MM-R", indicating that iterative reason- ing contribute to enhanced performance. We present our findings the FVQA dataset in Ta- ble 5. Notably, FVQA (Wang al., 2017) utilized the ground truth supporting fact triplets included in the dataset. contrast, MM-Reasoner surpasses this baseline without relying on these triplets. Fur- thermore, our framework outperforms the Mem- Net implementation by Shah et al. (2019), which achieved 59.2 accuracy on the KVQA test set, by +2.2 points, resulting in 61. 61.4 accuracy. It is important to note that MemNet uses ground truth entity names recognize celebrities in KVQA, whereas we employ an APT for this task.
cooateat": "Binchicy Wocd", 'contidence":
9,961, 1 Mscotentt's "CDURCII OOBIAMM" "conEldeuce""
[.391]
What country is this bus fzom?
sationales: Tho bus has Hinchley Wood and church Coanam writton
which ard located
angland.
cT: england, MiM+s angland, MM-R-3 E5, MM-
Reasomar: angland
"DeMer CAPTIONS's Meonntent's "men in mulla sitting table', 'contideace's 0yd14, "bourding_bon 640, 459)1, waz
0: What ethnicty this
nationates: The inage does not provida anough intormation detarmine stanicity the man.
white, MiMes unknowm R- whita, maanonem: white
Model
Acc
Figure 5: VQA examples on OK-VQA test set. GT refers to ground truth, For LLM+ and MM-R see 4.2.
PICa (Yang et al., 2022a)
48.0
KAT (Ensemble) (Gui et al., 2021)
54.4
Unified-io (2.8B) (Lu et al., 2022)
54.0
significance of vision APIs in framework.
REVIVE (Lin et al., 2022)
56.6
Flamingo (80B) (Alayrac et al., 2022)
57.8
4.3 Results and Discussion
REVIVE (Ensemble) (Lin et al., 2022)
58.0
REVEAL-Large (Hu et al., 2022b)
58.0
Our results on OK-VQA and A-OKVQA datasets are shown in Table 2 and Table 3, respectively. MM-Reasoner outperforms all baseline models. including the state-of-the-ar models PromptCap, Flamingo, and REVIVE on both datasets. More- over, the few-shot setting, MM-Reasoner signifi- cantly surpasses the state-of-the-art Flamingo-9B by considerable difference of 4.3 points (Table 4). This demonstrates that MM-Reasoner can integrate the strengths of both LLMs and VLMs to answer complex visual questions. Figure 5 shows examples. MM-Reasoner can find the answer to the first challenging question using OCR API. In the second example, LLM+, which leverages GPT- 4 and vision APIs, cannot find the answer since
REVEAL (Hu et al., 2022b)
59.1
PromptCap (Hu et al., 2022a)
60.4
MM-Reasoner
59.2
MM--Reaooner (Ensemble)
60.8
Table 2: Performance on OK-VQA test set.
4.4 Implementation Details
GPT-4-32k and Vicuna-13B (Chiang et al., 2023) from FastChat are used as the LLM in our frame- work. GPT-4-32k extends the context-length 32,000 tokens compared to GPT-4. The maximum
VQA
Phttps://githuu cow/lmmsyseeeeeeeeoeee ltt.///ssy///athatt

---

Model
Acc
initialization and select the final answer based on majority voting. For Flamingo-9B, shots is used and cross attention applied after every layers.
ClipCap (Schwenk et al., 2022)
25.9
ViLBERT (Schwenk et al., 2022)
25.9
LXMERT (Schwenk et al., 2022)
25.9
KRISP (Schwenk et al., 2022)
27.1
GPV-2 (Schwenk al., 2022)
40.7
Unified-1O (Lu et al., 2022)
45.2
REVEAL (Hu et al., 2022b)
52.2
Prophet (Shao et al., 2023)
55.7
PromptCap (Hu et al., 2022a)
59.6
MM-Reasoner (Ensemble)
60.2
Table Performance on A-OKVQA test set.
'DENGX CAPTIONS's [f [''oontent's "a plate of Food with fork", "Toonf denno 0.58), "bounding hoo' 649, 649]), exs
token context-length for Vicuna-13B is 2,048, but it is much faster than GPT-4. We use GPT-4-32k for experiments on OK-VQA, A-OKVQA, and FVQA, while Vicuna- 13B is used for experiments on the KVQA dataset since it is significantly larger than other knowledge-based VQA datasets. The LLM module remains frozen in all of our experi- ments. The temperature and top_p are both set 0.98. Additionally, frequency_ -eeaanlyyyppeeyyyyyeeeee and presence_penally are both set 0. The number in-context examples, k. is set to 20 for GPT4- 32k and 2 for Vicuna. We utilize APIs from the Image Analysis SDK version v4.0, except for the celebrity recognizer. The celebrity recognizer API is used for the KVQA dataset only.
the
called?
rimal Hiarsiions
Bationsles: Theze is clear indication of red iten.
the
tenato, LiM+: unable desermine, 18t-Reasoacr: cabbage: C.b, kade: 0.25, tonatos 0.25
Second iteration:
Rationa was
ned, the moat
1i anower 1a tomato..
LAMI: tonato, MM Reasoners tonatos 1.0
Figure 6: MM-Reasoner can refine its answer via it- erative reasoning. In the second iteration, the VLM answers are added the prompt and fed into the LLM to generate updated rationales for the VL.M.
Method
Acc
We use i-Code v2 (Yang et al.. 2023b) pre- finetuned on the VQA v2.0 dataset (Goyal et al.. 2017) and Flamingo-9B (Alayrac et al., 2022) from OpenFlamingo (Awadalla et al., 2023) 4 the VLM in our framework. We utilize Flamingo-9B only for few-shot experiments, while i-Code v2 is used in all other experiments. The i-Code v2 is fine-tuned during training, while Flamingo-99 remains frozen. The i-Code v2 prompt is "Answer the following question based on the given image: <Question> Hint: "Rationales>" Note that we use voca-voca VLM. All fine-tuning jobs are conducted on two RTX A6000 GPUs in less than few hours. The learning rate, batch size, and number of epochs are set to 2.0 10 6 4, and respectively. The number of VLM instances, m, is set to 4. For ensemble learning, we fine-tune 32 instances of the entire framework with different
Flamingo-9B (Alayrac et al., 2022) MM-R
35.7
40.0
Table Few-shot performance on OK-V VQQ.
Method
Acc
Human
77.99
FVQA (Wang et al., 2017)
56.91
ZS-VQA (Chen al., 2021)
58.27
FVQA (Ensemble) (Wang et al., 2017)
58.76
RM-Reasoner (Ensemble)
61.1
Table 5: Performance FVQA dataset.
5 Conclusion
MM-Reasoner is a new, unified framework for KVQA that combines vision APIs, the capabilities of LLMs, and VLMs. Experiments showed MM- Reasoner is an effective framework for KVQA.
htttps://github. http://c.oommmooooooeoeommmm
htttps://githbb com/n Foundations/open. flamingo

---

Limitations
visual question answering using knowledge graph. In The Semantic Web-ISWC 2021: 20th International Semantic Web Conference, ISWC 2021, Virtual Event, October 24-28, 2021, Proceedings 20, pages 146- 162. Springer.
Although the MM-Reasonel framework demon- strates strong performance in knowledge-based vi- sual question answering tasks, it has some limita- tions worth mentioning:
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang. Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez. et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https:/vvcuna. Imsys. org (accessed 14 April 2023).
Frozen LLM: The LLM used in MM-Reasoner remain frozen throughout the experiments, mean- ing it is not fine-tuned during training. This might limit the LLM's potential to adapt to the specific downstream tasks or evolving datasets.
Leveraging Language APIs and Tools: The MM- MM-Reasoner framework can be extended to in- corporate language APIs and tools, such search, translator, and calendar, to answer questions that require up-to-date information or time-related con- text. By integrating these APIs and tools, the frame- work can access the latest information and adapt its reasoning process accordingly. For example, consider the following QA scenarios that require the latest information:
Yash Goyal, Tejas Khot, Douglas Summers-Staa Dhruv Batra, and Devi Parikh. 2017. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913.
Liangke Gui, Borui Wang, Qiuyuan Huang, Alex Haupt- mann, Yonatan Bisk, and Jianfeng Gao. 2021. Kat A knowledge augmented transformer for vision-and- language. arXiv preprint arXiv:21
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah Smith, and Jiebo Luo. 2022a. Prompteap: Prompt-guided task-aware image captioning. arXiv preprint 22v:2211100999
Question: Who won the most recent edi tion of the tournament in the image? A search API can be useful here to retrieve the latest winner of the tournament, as this information might not be present in the LLM's pre-existing knowledge.
Ziniu Hu, Ahmet Iscen, Chen Sun. Zirui Wang. Kai- Wei Chang, Yizhou Sun, Cordelia Schmid, David A Ross, and Alireza Fathi. 2022b. Reveal: Retrieval- augmented visual-language pre-training with multi- source multimodal knowledge memory. arXiv preprint
Question: When is the next public holiday represented in the image? A calendar APT can provide the necessary information about up- coming public holidays, allowing the framework to generate an accurate answer based on the current date and the holiday calendar of the given coun- try. By integrating such language APIS and tools, MM-Reasoner can enhance its ability to handle a broader range of questions that require up-to-date information or specific time-related context, result- ing in improved performance in knowledge-based visual question answering task.
Yuanze Lin, Yujia Xie. Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. 2022. Revive: Re- gional visual representation matters in knowledge- based visual question answering. arXiv preprint arXiv:2206.01201.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. 2022. Unified- io: A unified model for vision, language, and multi- modal tasks. arXiv preprint 27X.v.2206.08916.
References
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. Advances in Neura! Information Processing Systems, 35:23716-23722
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. 2019. Ok-vqa: A visual ques- tion answering benchmark requiring external knowl- edge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pages 3195 3204.
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hes- sel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, et al. 2023. Openflamingo.
Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014, Glove: Global vectors for word rep- resentation. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543.
Zhuo Chen, Jiaoyan Chen, Yuxia Geng, Jeff Pan, Zonggang Yuan, and Huajun Chen. 2021. Zero-shot

---

Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer. Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: tools.
2023b. i-code v2: An autoregressive generation framework over vision, language, and speech data. arXiv preprint arXiv: 230111311..
Language models can teach themselves arXiv preprint urXiv:2302.04700
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vin- cent Vanhoucke, et al. 2022. Socratic models: Com- posing zero-shot multimodal reasoning with lan- guage. arXiv preprint 20X.0:2204.05598
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: benchmark for visual question answering using world knowledge. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII, pages 146-162. Springer.
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Talukdar. 2019. Kvqa: Knowledge- aware visual question answering. In Proceedings of the AAAI conference on artificial intelligence, vol- ume 33, pages 8876-88 8884.
Zhenwei Shao, Zhou Yu, Meng Wang, and Jun Yu. 2023. Prompting large language models with answer heuris- tics for knowledge-bard based visual question answering. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pages 14974- 14983.
Peng Wang, Qi Wu, Chunhua Shen, Authony Dick, and Anton Van Den Hengel. 2017. Fyqa: Fact-based visual question answering. IEEE transactions pat- tern analysis and machine intelligence, 40(10):2413- 2427.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yu- lia Tsvetkov, and Yuan Cao. 2021. Simvim: Simple visual language model pretraining with weak super- vision. arXiv preprint arXiv:2 10904.
Jialin Wu, Jiasen Lu, Ashish Sabharwal, and Roozbeh Mottaghi. 2022. Multi-modal answer validation for knowledge-basee vqa. Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2712-2721.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang. Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022a. An empirical study of gpt-3 for few-shot knowledge- based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 308 1- 3089.
Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023a. Mm- react: Prompting chatgpt for multimodal reasoning and action. arXiv preprint 2303. 11811.
Ziyi Yang, Yuwei Fang, Chenguang Zhu, Reid Pryzant, Dongdong Chen, Yu Shi, Yichong Xu, Yao Qian, Mei Gao, Yi-Ling Chen, et 2022b. i-code: An integra- tive and composable multimodal learning framework. arXiv preprint ar..vv2205.01818
Ziyi Yang, Mahmoud Khademi, Yichong Xu, Reid Pryzant, Yuwei Fang, Chenguang Zhu, Dongdong Chen, Yao Qian, Mei Gao, Yi-Ling Chen, et al.


Reviews:
[{"review_id": "PlDw0TgkCi", "reviewer": "Reviewer_wKib", "summary": "", "questions": "- Question A: The number of in-context examples is 20 for GPT-4 and 2 for Vicuna. How did you set the number of examples? The variance of the number seems to be high.\n- Question B: This paper adopts 4 number of VLM instances (described in L545). Have you conducted an ablation study on the number of VLM instances? Multiple VLM instances are necessary?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments. Regarding the experimental results, we have indeed reported the performance of a single model (MM-R) in Table 1 (Acc=59.2). MM-R represents a single MM-Reasoner framework without the use of ensemble learning (as mentioned in Line 453). It is important to note that our single model performance surpasses that of state-of-the-art models such as REVIVE (Ensemble), Flamingo (80B), REVEAL-Large, and REVEAL, as shown in Table 2. To facilitate a more straightforward comparison, we will add the MM-R single performance to Table 2 as you suggested. Also, in the case of few-shot experiments, we employed another single MM-Reasoner framework (MM-R^o), which significantly outperforms Flamingo (80B) as presented in Table 4.\n\nFor the few-shot experiments, Flamingo-9B was utilized (referenced in Line 535). We employed 8 shots for Flamingo-9B and applied cross attention after every 4 layers (as stated in Line 549). All other hyperparameters were set to the default values provided in the open_flamingo Github repository (https://github.com/mlfoundations/open_flamingo). We will ensure that these details are clearly articulated in the paper and will also include them on the MM-Reasoner Github page.\n\nIn response to Question A regarding the varying number of in-context examples for GPT-4 and Vicuna: The primary reason for setting the number of in-context examples is based on the observation that performance often improves when the number of in-context examples increases. Consequently, we determined the number of in-context examples according to the maximum token context-length permitted by each model. For GPT-4-32k, the maximum token context-length is 32,000, while for Vicuna-13B, it is limited to 2,048 (as stated in Line 514). Due to this constraint, Vicuna-13B can only accommodate 2 in-context examples.\n\nIn response to Question B regarding the use of 4 VLM instances: The primary motivation for employing multiple VLM instances is to facilitate Iterative Reasoning by examining a range of candidate answers instead of solely focusing on the highest-confidence response. While utilizing 4 VLM instances (m=4) is one approach to achieve this, alternative methods such as incorporating a beam search decoder (referenced in Line 327) could also be employed. Comparative analysis revealed that using 4 VLM instances improved performance by approximately 0.7 points compared to a single VLM instance (m=1). However, increasing the number of instances beyond 4 (m>4) did not yield any further improvements.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The topic discussed in this paper and the idea of the proposed approach leveraging LLM, VLM, and additional plugins would draw attention in the NLP community and well suits the EMNLP conference.\n- The idea of this paper is simple but meaningful. It explores not only LLMs with the plugins but also cooperation between an LLM and a VLM. In addition, further improvements in iterative reasoning and similarity-based in-context learning also be interesting and would promote further related studies.", "reasons_to_reject": "- The experimental results are somehow weak. The performance improvement seems to be marginal. Especially, the only ensemble results are reported in Tables 2, 3, and 5. The result of using a single model should be reported for a fair comparison.\n- The details about the few-shot experiment are almost missing. The motivation and the implementation details should be explained.", "questions_to_authors": "- Question A: The number of in-context examples is 20 for GPT-4 and 2 for Vicuna. How did you set the number of examples? The variance of the number seems to be high.\n- Question B: This paper adopts 4 number of VLM instances (described in L545). Have you conducted an ablation study on the number of VLM instances? Multiple VLM instances are necessary?", "paper_topic_and_main_contributions": "* This paper proposes a multi-modal knowledge-aware framework (MM-Reasoner) exploiting a large language model and a vision-language model. MM-Reasoner consists of a set of vision APIs, an LLM, and a VLM, and these components cooperate to predict an answer to a given question by extracting and exchanging the intrinsic and extrinsic knowledge about a given image and the question.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ASBqOhoHQ4", "reviewer": "Reviewer_XP8j", "summary": "", "questions": "In Eq 1, how is Y_ic obtained?\nIn Eq 2, what does V and Q denote? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments and suggestions. Regarding the cost and efficiency, all state-of-the-art frameworks utilize LLMs, such as GPT-3, and train with vast knowledge resources and VQA datasets to address this challenging task. Consequently, both the training and evaluation processes are highly expensive in these frameworks. In contrast, MM-Reasoner, as a modular and integrative framework, is compatible with a wide range of existing APIs, LLMs, and VLMs, including open-source options, Hugging Face models, and Azure Cognitive Services/APIs. Since these LLMs/VLMs have already been trained on extensive datasets, the fine-tuning process for MM-Reasoner is significantly less costly compared to other cutting-edge approaches. In fact, all fine-tuning tasks can be accomplished on two RTX A6000 GPUs in just a few hours (Line 541). Additionally, the modular and composable architecture of MM-Reasoner allows it to benefit from advancements in knowledge distillation methods. These methods can reduce the model size of LLMs and VLMs, further enhancing the efficiency of MM-Reasoner.\n\nRegarding the prompt construction, we appreciate the opportunity to clarify the details of our prompt construction process. In-context examples are selected from the training data, with the A-OKVQA dataset providing ground-truth rationales for answers in the training set (Line 393). For the OK-VQA dataset, we manually created rationales for a small subset of in-context examples, which will be made available on the MM-Reasoner Github page. This small subset serves as the basis for our dynamic in-context example set, tailored to each image and question pair as described in the k-NN-based in-context examples subsection (Line 362).\n\nThe number of in-context examples, k, is set to 20 for GPT-4-32k and 2 for Vicuna (Line 525). For GPT-4-32k, the maximum token context-length is 32,000, while for Vicuna-13B, it is limited to 2,048 (Line 514). Due to this constraint, Vicuna-13B can accommodate only 2 in-context examples.\n\nRegarding Figure 6: We appreciate your suggestion and will move Figure 6 to Section 3 in the final version, as the extra page allows for it. Additionally, the prompts will be shared on the MM-Reasoner Github page.\n\nFor Eq 1, the ground-truth rationale for the in-context example X_ic, denoted by Y_ic, is obtained as follows - for the A-OKVQA dataset, the Y_ic is provided by the creators of the dataset for each training and validation samples (Line 393). For the OK-VQA dataset, we manually created the ground-truth rationales for a small subset of in-context examples. An example of a rationale for the OK-VQA dataset can be found in Figure 6. We will make these rationales available on the MM-Reasoner GitHub page. As for Eq 2, Q represents the input question, while V denotes the given visual input or image (Line 299).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The method is novel in utilizing vision APIs to extract various features and inputting this information into an LLM to get query-specific knowledge. This operation makes a good use of the LLM and can obtain richer information, external knowledge, commonsense, explicit supporting facts, and rationales.\n2. Experimental results show that the proposed method outperforms current approaches and achieves SOTA performance.\n", "reasons_to_reject": "1. The framework is quite a big one which uses many external APIs and an LLM, which may make it difficult to deploy. And what is the cost and efficiency?\n2. Some details in the process of prompt construction may be made clearer. For example, how are the examples selected or constructed? How many examples are used for each sample? Also, I think Figure 6 is very important for understanding the method, however, it is in the Appendix due to page limit and may be not seen by the readers when the paper is published. Is it possible to add some key information of Figure 6 into the paper?\n", "questions_to_authors": "In Eq 1, how is Y_ic obtained?\nIn Eq 2, what does V and Q denote? ", "paper_topic_and_main_contributions": "This paper proposes a new framework called Multi-Modal Knowledge-Aware Reasoner (MM-Reasoner) for KVQA, which utilizes a set of vision APIs to extract detailed information from the image in textual format and then prompts an LLM to extract query-specific knowledge from the extracted textual information to provide a rich representation. Finally, the knowledge, query, and visual input are used to fine-tune a Vision-Language Model (VLM). Empirical studies show that MM-Reasoner achieves state-of-the-art performance on several KVQA datasets.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E8w1R5a0Xr", "reviewer": "Reviewer_tUjL", "summary": "", "questions": "1. Why do we need to use an additional vision language model? There are two heavy large language models used in this work. And I think it makes no sense to need two complex models to do duplicated things.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful comments. As we discussed in the introduction, the KVQA task involves a diverse set of questions, some requiring extensive external knowledge, language understanding, and reasoning capabilities, while others necessitate understanding visual concepts such as the checkered pattern of a table cloth. To address this diversity, our proposed MM-Reasoner aims to provide (1) vision expertise, e.g., celebrity recognition, (2) the reasoning capacity and world knowledge of recent LLMs, and (3) the ability of VLMs to generate rich joint representations of vision and language within a unified framework. This approach allows our model to outperform state-of-the-art performance on various diverse datasets. Another key motivation to use both LLMs and VLMs is that LLMs are trained using text data like Wikipedia and books, whereas VLMs utilize image-caption pairs for training. This distinction in pre-training domains results in differing, and occasionally complementary, abilities for LLMs and VLMs. For example, LLMs are well-suited for questions requiring reasoning capabilities, but they have difficulty understanding visual patterns. MM-Reasoner capitalizes on the strengths of both LLMs and VLMs (as noted by Reviewer wKib) to tackle the KVQA task.\n\nAn alternative method would be to train a model with massive datasets of image-text pairs plus massive text-only datasets, as in GPT-V (GPT-4 with visual input). However, this approach is more expensive, and the model may not be easily accessible. Furthermore, the concept of adding tools and plugins remains valuable for models like GPT-V, as it increases the capabilities of the model and reduces the number of training examples required to train the model. Experiments show that GPT-V sometimes fails to recognize celebrities, for instance.\n\nIn comparison to state-of-the-art frameworks that utilize LLMs like GPT-3 and require highly expensive training and evaluation processes, MM-Reasoner offers a more cost-effective solution. Its modular and integrative framework is compatible with a wide range of existing APIs, LLMs, and VLMs, including open-source options, Hugging Face models, and Azure Cognitive Services/APIs. These LLMs/VLMs have already been trained on extensive datasets, making the fine-tuning process for MM-Reasoner significantly less costly. In fact, all fine-tuning tasks can be completed on two RTX A6000 GPUs in just a few hours (Line 541).\n\nMoreover, MM-Reasoner's modular and composable architecture allows it to benefit from advancements in knowledge distillation methods, which can reduce the model size of LLMs and VLMs and further enhance the efficiency of our proposed framework.\n\nRegarding the format: We will improve the format and move Figure 6 to section 3, as the extra page allows for it.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. They consider the VQA problem in the ChatGPT area\n2. They used the visual recognized result to enhance the context information with a generative model", "reasons_to_reject": "1. The overall framework is heavy and not that makes sense.\n2. The text and formatting needs to be improved.", "questions_to_authors": "1. Why do we need to use an additional vision language model? There are two heavy large language models used in this work. And I think it makes no sense to need two complex models to do duplicated things.", "paper_topic_and_main_contributions": "This paper presents MM-Reasoner, a new framework for knowledge-based visual question answering (KVQA) tasks. The system combines the strengths of large language models (LLMs), vision APIs, and vision-and-language models (VLMs) to handle complex questions that require external knowledge and fine-grained visual understanding.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Null"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces MM-Reasoner, a framework designed for knowledge-based Visual Question Answering (VQA) that harnesses the strengths of multiple models and tools. Specifically, when presented with an image and a question, MM-Reasoner engages vision APIs, an LLM, and a vision-language model in tandem to address the query.

All reviewers commend the innovative approach of amalgamating the capabilities of various models and plugins. The method demonstrates commendable empirical results on the OKVQA benchmark. However, some reservations exist. Two reviewers note that the framework is intricate, necessitating intensive engineering of the pipeline. They also highlight concerns about the assumed availability and access to a multitude of tools/plugins, such as APIs.
Decision: Accept-Findings