MoqaGPT : Zero-Shot Multi-modal Open-domain Question Answering with Large Language Model
Authors: Le Zhang
Yihong Wu
Fengran Mo
Jian-Yun Nie
Aishwarya Agrawal
Venue: EMNLP 2023 Findings (2023)
Keywords: Large Language Model
Multimodal
Open-domain question answering
TL;DR: We propose a Zero-Shot Multi-modal Open-domain Question Answering framework based on LLMs and achive good performance on downstream tasks

Abstract: Multi-modal open-domain question answering typically requires evidence retrieval from databases across diverse modalities, such as images, tables, passages, etc. Even Large Language Models (LLMs) like GPT-4 fall short in this task. To enable LLMs to tackle the task in a zero-shot manner, we introduce MoqaGPT, a straightforward and flexible framework. Using a divide-and-conquer strategy that bypasses intricate multi-modality ranking, our framework can accommodate new modalities and seamlessly transition to new models for the task. Built upon LLMs, MoqaGPT retrieves and extracts answers from each modality separately, then fuses this multi-modal information using LLMs to produce a final answer. Our methodology boosts performance on the MMCoQA dataset, improving F1 by +37.91 points and EM by +34.07 points over the supervised baseline. On the MultiModalQA dataset, MoqaGPT surpasses the zero-shot baseline, improving F1 by 9.5 points and EM by 10.1 points, and significantly closes the gap with supervised methods. Our codebase is available at https://github.com/lezhang7/MOQAGPT.

MOQAGPT: Zero-Shot Multi-modal Open-domain Question Answering with Large Language Models
Le Zhang 122 Yihong Wu', Fengran Mo², ..iin----uu Nie², Aishwarya Agrawal!.2
Mila Québec AI Institute
Université de Montréal
Zhang, aishwarya aww.a/a..aaaaaaaaaaaa Jmmi Jeuebe.uuebeccccccccccmmmooomo
Abstract
Quession
Where the movie The Shape Water made?
Multi-modal open-domain question answer- ing typically requires evidence retrieval from databases across diverse modalities, such images, tables, passages, etc. Even Large Lan- guage Models (L.LMs) like GPT-4 fall short in this task. To enable LLMs to tackle the task in zero-shot manner, we introduce MOQAGPT', a straightforward and flexible framework. Using a divide-and-coqquer strategy that bypasses in- tricate multi multi-modalityy ranking, our framework accommodate
Retrieve
Preeae !1 Two
make 0.200
Maal 21
Paaaage 4
Plssage Fineng
2013
Ediianne
Reasoning
Jeth The mook Torees
sgge as Eanads A A'sogt Canada
A puc The ofagpnaa
et Leannsonn Suator bigkind A Lbb lvencc
Thinking
ALiig& U2
A_40C Semiso
modalities and seam- lessly transition to new models for the task. Built upon LLMs, MOQAGPT retrieves and ex- tracts answers from each modality separately, then fuses this multi-modal information using LLMs to produce a final answer. Our method- ology boosts performance the MMCoQA dataset, improving Fl by +37.91 points and EM by +34.07 points over the supervised baseline. On the MultiModalQA dataset, MOQAGPT surpasses the zero-shot baseline, improving Fl by 9.5 points and EM by 10.1 points, and signif- icantly closes the gap with supervised methods.
Carsichies Releevd Insee??
Toronio
Figure 1: An illustration how human adopt divide- and-conquer strategy to answer multimodal open- domain question
the accuracy and reliability of question answering, as it can introduce plausible, yet incorrect informa- tion, thus exacerbating the problem. Second, while LLMs are designed to process the text modality only, there numerous other non-textual sources of information, such as images, videos, audios, and tables, that could provide suitable answers to most real-world questions. Some queries may even require a synthesis of information from across dif- ferent modalities for accurate responses. Thus, the inability to process con-textual inputs restricts the effectiveness of current LLMs.
1 Introduction
Large Language Models (LLMs) including Chat- GPT (OpenAI, 2022b), LLaMA (Touvron al.. 2023), P2LM2 (Anil et al., 2023), and the recently developed GPT4 (OpenAI, 2022c), have fundamen- tally transformed the manner in which humans in- teract with machines. Due to their vast knowledge repositories and chain-of-thought reasoning capa- bility (Wei et al., 2023), these models have proven to be capable of providing answers to broad range of questions across domains, without the need for training on specific tasks. Nevertheless, these mod- els face two significant challenges. First the issue hallucination (Li et al., 2023), attributable to the fact that LLMs store their knowledge in their parameters. Hallucination can seriously hamper
Consider an example outlined in fig. where the question asked is, "Where was the movie 'The Shape of Water' made?". To effectively answer this question, human would employ a divide- and-conquer approach. This strategy involves first retrieving relevant documents such as the movic poster, news reports, and table records for The Shape of Water. The individual would then de- rive the answer from the obtained references that might include terms like Canada, Toronto, or Lon- don, Hertfordshire in England. Comprehensive rea-
'Our codebase is available at https://gitthh com/ lezhang7/HOgAGPT

---

Old Supervised Paradigm
Zero-shot Powered by LLM
Mull-mmdisily Guestion Answer
Information ExPact
Special
System
LLM information
Muliimadaaity Ranker
Image tinosder
Toxt Fintzater
Table
Quessioo
Moddlltyiindependentty Retrieval
Einradw
Ennender
Training File
Large Mulf- model Open-damein Arnotations Lange Mull-omdial Aligament Datisell Special Anniaaionns Graph, Reislion, Question Type Amnotations
task specifie Annotations needed anymore Add as much modality as possible!
Figure 2: Comparison of two paradigmns for multimodal open-domain question answering. Fire symbol indicates modules require training, ice symbol indicates forzen models.
soning would be applied to all potential answers, for instance, recognizing that Toronto is related Canada. Given this relationship, and the lack of strong ties between London, Hertfordshire in England and other candidate answers, Toronto is deemed the most probable answer and selected as the final response.
fig. 2, has three main advantages: Flexibility: The MOQAGPT operates in zero-shot mode without re- lying joint representation inference, which al- lows easy replacement of individual modules with more advanced ones as they become available. Fur- thermore, it can accommodate a wider range of modalitics and climinates the need to curate a mul- timodal open-domain dataset for training. Trust- worthiness: The framework's 's responses are based on the retrieved results. Consequently, each an- swer can be traced back to its original source, thus making the model more trustworthy and reducing the risk of hallucination. Interpretability; All in- termediate outputs, including the retrieval results, candidate answers, and the final reasoning for an- swer synthesis, are produced in natural language, rendering the process of answering open open- nom-nnomain questions transparent and interpretable.
However, many existing models rely on a joint strategy (Liu et al., 2023; et al., 2022; Yang et al., 2022; Chen et al., 2022) where they attempt to re- trieve and rank all modalities by training joint embedding space. This approach, despite its appli- cation, has several shortcomings. Firstly, the joint strategy lacks flexibility, necessitating retraining when new models and modalities are introduced. Secondly, it poses considerable difficulties to train a joint embedding space and rank references en- compassing more than two modalities. Although some sovide-and-conquee models have been pro- posed (Talmor et al., 2021), they also come with limitations. These models require training ques- tion type classifier, followed by the training of var- ious guestion-answering models. Each of these stages requires a significant amount of annotations, thus presenting a considerable challenge in their development and implementation.
We corroborate these advantages by present- ing experimental results on two multi-modal open- domain question answering (MMOQA) datasets: MMCoQA (Li et al., 2022) and MultModalQA (Talmor al., 2021). Both datasets require ques- tions to be answered based on information retrieved from text, image, and table references. We con- ducted experiments using several of the latest mod- els and demonstrated that our method is effective across all of them, highlighting our framework's ss flexibility. To demonstrate the trustworthiness of our method, we compared our outputs to those pro- duced by directly querying LLMs. Our outputs are less prone to hallucination, making them more trustworthy. Lastly, we examined several success and failure cases of our method. Thanks to the inter- pretable nature of our framework, we could identify the sources of errors. Overall, our method exhibits
To enable LLMs to solve this task in a zero-shot manner, we propose Multi-modal Open-domain Question Answering GPT (MoQAGPT). Mo- QAGPT utilizes a divide-and-come conquerr approach and employs robust models to extract answers from various modalities. It further leverages LLMs reasoning mechanism, applying in-context learning to process the extracted information and generate the final response. Compared with the traditional supervised methods, our framework, as depicted in

---

robust zero-shot performance on both datasets, un- derscoring its promising potential.
like (Andreas et al., 2016, 2015) aimed to handle compositional tasks. They did this by decompos- ing them into sub-tasks, utilizing off-the-shelf lan- guage parsers, and then learning specialized neural modules for each. However, their applicability was limited, being constrained by parser performance and the need for hand-specified module types.
Our contributions this paper are threefold: (1) We propose MOQAGPT, a simple and effective framework, which is the first to enable L.MMs to tackle multi-moda open-domain queries in zero- shot setting, (2) We conduct extensive experiments involving multiple LLMs and Vision-Language Models (VLMs), thus validating the effectiveness of our approach. (3) We present empirical evidence that LLMs are capable of efficiently addressing MMOQA tasks when paired with other modali- ties. Furthermore, we demonstrate that replacing each module with its superior version enhances performance, establishing this as foundational framework for future zero-shot question-answering systems.
The emergence of LLMs has renewed interest in this area. LLMs address the parsing challenges without necessitating additional training. Conse- quently, this led to the proposition of various LLM- based systems targeting an array of compositional reasoning challenges. Examples include:
Toolformer (Schick et al., 2023), which trains language models select tools. Visual ChatGPT (Wu et al., 2023), HuggingGPT (Shen et al., 2023). and Chameleon (Lu et al., 2023), all utilizing GPT to deduce tool sequences for response generation. ViperGPT (Surís et al., 2023) and Visprog (Gupta and Kembhavi, 2022), leveraging Codex and GPT3 (Brown et al., 2020) respectively to produce python programs for visual reasoning tasks. Yet. these methods don't address MMOQA. MMOQA inher- ently involves multiple steps, making apt for a modular approach. Our framework, therefore, cap- italizes on LLMs to integrate and reason about information retrieved from various modalities for MMOQA. Our approach is distinct as it requires no training, setting it apart from prior works like Schick et al. (2023). It also differentiates itself from research such as Surís et al. (2023); Gupta and Kembhavi (2022); Shen et al. (2023) with its emphasis on retrieval-based question answering. Although Chameleon (Lu et al., 2023) supports both retrieval and question answering, it doesn't 't address the MMOQA tasks, especially those need- ing cooss-modal information integration and rea- soning. Morcover, method operates in zero- shot setting, avoiding the need for intermediate programs, unlike Chameleon which requires a few- shot Python intermediate program.
2 Related Work
Multi-modal Open-domain QA MMOQA rep- resents a challenging yet realistic task that is cru- cial for all future automated question-answering systems. This task necessitates the retrieval of pertinent references, after which answers are ex- tracted from these references. This often involves complex processes such as modality selection and cross-modal reasoning. In light of this, several datasets have been introduced to benchmark the de- velopment of solutions in this area, such as Many- ModalQA (Hannan et al., 2020), HYBRIDQA (Chen et al., 2020), WebQA (Chang et al., 2022), MultiModalQA (Talmor et al., 2021), and MM- CoQA (Li al., 2022).
Earlier works this field focus on model train- ing. These include methodologies for joint embed- ding of multiple modalities (e.g. MAE (Li et al., 2022) and ManeModeIQA (Hannan et al., 2020)), the structured knowledge and unified retrieval- generation based method SKURG (Yang et al.. 2022), and the Multimodal Graph Transformer (He and Wang, 2023), which employs a graph- based quasi-attention mechanism for integrating multi-modal graph information. To the best of our knowledge, we are the first to introduce a zero-shot method for multi-modal open-domain question an- swering, marking a significant contribution.
MOQAGPT
MOQAGPT presents a general approach to gener- ate answers to queries using a multi-modal knowl- edge base collection C, which encompasses text Ctut, tables Crab, and images Cing. The divide- and-conquer strategy is accomplished through a two-stage process.
LLM-based Modular Systems The develop- ment of modular neural networks, rooted in biol- ogy and neuroscience, can be traced back to the 1990s (Azam, 2000; Auda and Kamel, 1999). Be- fore the rise of LLMs, modular neural networks
First, the Multi-modal Question Answer Extrac- tion stage ($.3.1 I) extracts answer candidates from

---

Stage1: Zero-shot Multimodal Question Answer Extraction
Stage2: Zero-shot Answers Infusion
Image Retriever
Visual QA
Answer Candidates [A_ IMG, A_TEXT, A, TAB, APGTT
Image KB
VLMs
Rule-based Strategy
Prompt
Given the question (Question), please select the best from the following candidates (Answer Candidates):
Text Retriever
Text &Table QA
Question
Text KB
LLMs
GPTs Reasoner
Table Retriever
Direct DA
Re-extract answer
Table KB
GPTs
Final Answer
Figure 3: Overview of MOQAGPT. Snow symbol indicates the model is frozen.
different modalities of the knowledge base inde- pendently and utilizes rule-based strategies to sift through these responses. Second, the Answers In- fusion stage ($3.2) employs LLMs' reasoning abil- ities to integrate information from across various modalities and select the most plausible answer. noteworthy that MOQAGPT operates with ex- isting models and requires no additional training, thus exhibiting zero-shot inference capabilities. A comprehensive depiction of this methodology is provided in fig. 3, and the prompts used are de- scribed in table 1. It's worth noting that we do not perform prompt engineering due to the API cost incurred.
pressive performance in image-text retrieval bench- marks (Lin et al., 2014; Young et al., 2014). The similarity between queries and images for retrieval is determined through their inner product. (iii) For Table Retrieval, tables are typically converted into textual format, and then language models are utilized for encoding (Herzig al.. 2020), Fol- lowing this protocol, we employ OpenAl's Ada (OpenAI, 2022a), a robust embedding model, to encode linearized tables and queries, with similar- ity measured using the inner product. The retrieval process can be expressed as:
Rimg =
ImageRetriever(q, Cimg)
Res TextRetriever(q, Cir)) Riah = TableRetriever(g, linearize(Cras))
3.1 Multi-modal Question Answer Extraction
In this stage, queries are answered independently for each modality, and a strategy employed to re- fine the responses. It's important to emphasize that every retrieval modality and question-answent model within our framework is frozen and can be interchanged. We delve into the details of this stage in the subsequent sections.
where R represents the retrieved references, the question, and C is the knowledge collection.
3.1.2 Question Answering
Upon retrieving the references, the next step is to extract answer from each reference based on the question g. (i) For Visual QA, we use vision- language models (VLMs) with robust zero-shot capabilities to generate the responses. In this step, the question fed into these VLMs using a simple prompt: 'Question: Q Answer:'. (ii) Since table data can be linearized into text, both Textual QA and Tabular QA can be tackled with single LLM. As we' re aiming for extractive question answering, the final response should be concise text span from the provided input. We direct the LLMs with specific prompt (refer Prompt QA in table 1) for this purpose.
3.1.1 Retrieval
Different modalities utilize pre-established, highly effective models for retrieval, eliminating the need to map all modalities into a joint embedding space. (i) For Text Retrieval, we use the ANCE model (Xiong et al., 2020), which adopts a dense retrieval approach to encode both passages and queries. Re- trieval is then based the cosine similarity be- tween these encoded representations. (ii) For Im- age Retrieval, the renowned CLIP model (Radford et al., 2021) 1) is employed. This zero-shot retriever, trained on web-crawled cuption-image pairs us- ing an image-text contrastive loss, has shown im-
Moreover, we found that some questions could be addressed directly by prompting LLMs. Hence,

---

Name
Prompt
You are performing extractive question onswering. Given the document: (reference), extract a short answer the question: (Q) from the document. If insufficient
Prompt QA
information is available to answer the question, respond with 'Unknown'. The answer should be one or two words long.
Question: [questions]. Please provide concise response, limited or two words, No explanation and further question. Answer:
Prompt Direct-QA
Prompt Answer-Fusion
Given question (Q). please select the best answer from the following condidates: (Candidates)
Given the question (Q), please extruct the answer spun from [final answer), without providing additional sentences or explanations. The response should be single word.
Prompt Re-extract
Table I: Prompts used in MOQAGPT
we also incorporate answers from (ii) Direct QA, where responses obtained by directly querying the LLMs using a prompt (see Prompt Direct-QA in table 1). The overall process can be represented as:
These rules enforced in sequence. If rule I isn't satisfied, will have curated set of valid answer candidates, denoted as A Aimg: Atrt; A Adirect- This set will then be used to pinpoint the final answer span by the rea- soner, detailed in the subsequent section.
A img VLM(q, rlngg
3.2 Answer Infusion
Alstt LLM (Promppt riz))
For the majority of queries, even though it's hard to decide which modality contains the answer, the format of the answer is usually predetermined. For instance, a question like "What color is the Santa Anita Park logo?" should yield color as the an- swer, not a date or name. Inspired by this ob- servation, we leverage LLMs to infer the correct answer format and select the appropriate answer from the candidates. To achieve this, we designed prompt (refer to Prompt Iaswer-Fusion in table 1) that enables LLMs to determine the final answer. As the gold-standard answers are typically short, if the final answer contains more than three words, we guide LLMs to select the correct text span using the Prompt Re-extract.
Aia = LLM linearize(ria.))
Adirect DiLM(Poopp
where r² signifies the 1t reference from R.
3.1.3 Rule-based Strategy
At this stage, we possess answer candidates de- rived from various modalities. Nonetheless, the autoregressive generation of responses by LLMs and VLMs can sometimes produce invalid outputs, such "sorry, I can't 'tn ". Through empirical obser- vation, identified that: (i) The VLMs tend to consistently produce answers, even if relevant in- formation missing. (ii) The most accurate answer isn't necessarily found in the top-1 (most similar) retrieved reference. (iii) Using our prompts, LLMs can discern when to provide a specific answer and when to default to "unknown", especially when the available information is insufficient.
Experiments & Results
This section is organized follows. In $4.1, we outline MMOQA datasets, metrics, and baselines. We then discuss retrieval performance in $4.2, ques- tion answering in $4.3, and MMOQA results $4.4. Lastly, we present an ablation study in $4.5 followed by a detailed case study in $4.6.
With these insights, we crafted task-an aggnosicc rule-based strategy to filter out invalid spans and prioritize the most likely answers generated by the LLMs and VLMs: (1) If the direct answer, Adirect- is found within any of the sets Aimg. Aust Anah it's deemed reliable and is chosen as the final an- swer. (2) Any answer containing phrases like "un- known" or "sorry" is discarded. (3) Rather than exclusively relying on the top-1 retrieved reference, we choose the most frequent response from the top- K retrieved references. If all responses are distinct, we opt for the response from the top-1 reference.
4.1 Implementation details
Dataset and Metrics. We evaluate our method two MMOQA datascts (refer to table 2 for dataset statistics). Though they share the same references from tables, text, and images, they have different settings, questions, and answers. all our experiments, we utilize only the top references per modality from the knowledge collection.

---

Dataset
#Questions
#lmages
#Tables
#Texts
Modality
MRR
NDCG
Recall
Recall 2000
MMCoQA
590
57.058
10,042
218,285 218,285
Joint (NDCG@
MuliModallQA
2441
57,058
10,042
ORConvQA
2.3
19.1
1AE1
6.1
63,4
Table 2: Dataset statistics
Divide-and-Congae
The MMCoQA dataset (Li al., 2022) evaluates a model's proficiency in identifying the appropriate answer modality. Each question is uniquely tied to specific modality. While the dataset employs conversational structures with historical context, we only utilized the gold question for all test set experiments to emphasize non-conversational ap- proach.
Image (CLIP)
31.2
34.0
40.8
Tahle (Ada)
50
53.9
65.5
Text (ANCE)
35.6
39.0
49.0
Overall
51.0
Table 3: Retrieval results for MMCoQA | represents quoted results. Joint NDCG are computed for 2000 items. Ques- tions are classified into categories based on the gold reference modality, scores are computed for each modality indepen- dently. Overall result is computed on concatenated references from all modalities, which can be viewed as Recall @ 5x3.
The MultiModalQA dataset (Talmor et al., 2021) is designed for multi-modal comprehension and multi-hop reasoning QA. In alignment with prior studies (Li et al., 2022; Talmor et al., 2021; Yang al., 2022), we test the development set (the test set is unlabeled and online evaluation is available), using Exact Match and Fl as evaluation metries. The MultiModalQA dataset already pro- vides 1-15 reference candidates for each modality per question.
only. Each question has designated gold refer- ence modality, and we group questions based on this attribute to report a breakdown of results across modalities in table 3. This evaluation focuses solely on the retrieval of candidates.
The table indicates that our suride-and-conuuer approach provides significant benefits terms of effectiveness compared to the joint method. The prior state-of-the-ee methodology, MAE, trains knowledge encoders for tables, images, and tex- tual documents. Once trained, these knowledge encoders are frozen, and contrastive loss func- tion is employed to train the query encoder. This approach seek align the embedding spaces of ta- bles, images, and textual documents via the query embedding, without incorporating an actual multi- modality alignment. In contrast, our methodol- ogy disentangles intricate multimodal knowledge relationships by retrieving each modality indepen- dently and then assembling them using LLMs, elim- inating the need for complex ranking. It registers significant improvement, with its Recall @ 5x3 (51) being close MAE's Recall @ 2000 (63.4).
Baselines To the best of our knowledge, our ap- proach is the first to enable LLMs to perform zero- shot MMOQA. For zero-shot baselines, we select Direct QA by Vicuna, OpenChat, Llama2, Chat- GPT, and G Additionally, benchmark results against supervised methods: (i) For the MMCoQA dataset, we compare with the previ- ous state-of-the-art models, MAE (Li al., 2022), a joint embedding model trained on MMOQA datasets, and the ManyModelQA model (Hannan et al., 2020). These are the only models reported for this dataset. (ii) For the MultiModalQA dataset, we draw comparisons with the previous SOTA model SKURG (Yang et al., 2022), structured knowledge and unified retrieval ceneration-based method, and the Multimodal Graph Transformer (He and Wang, 2023), a model that employs a graph-based quasi-attention mechanism to inte- grate multi-modal graph information.
4.3 Question Answering Results
We conduct question answering on the retrieved references, obtaining 5 answer candidates for each modality. The primary assessment criterion is to determine the gold answers were present among these 5 candidates. As such, we use Recall @5 to evaluate the quality of the generated answers. Addi- tionally, Vicuna's 's outputs often appear to be noisy, lengthy, and non-specific. For instance, for ques- tion with the gold answer "Joss Whedon", Vicuna might produce response like "I. Joss Whedon 2. David Greenwalt 3. David Boreanaz 4. Unknown", which complicates the extraction of the final an- swer, even if the recall score is high. This recall
4.2 Retrieval Results
For the MultiModalQA dataset, previous works directly use the gold reference set without any retrieval. We perform retrieval on the provided reference candidates (1-15 per modality) and find that Recall @5 was consistently 100%. As re- sult, present the retrieval results for MMCoQA
Details are described in Appendix appendix B.

---

MMCoQA
MultiModalQA
VLM
LLM
Image
Table
Text
Overall
Single
Multiple
Overall
BLIP2
Vicuna
28.6
25.5
33.9
41.5
46.8
37.0
42.6
InstructBLIP
Vicuna
31.3
42.9
50.7
38.7
BLIP2
OpenChatV2
28.6
23.4
40.3
44.2
50.7
40.5
46.3
InstructBLIPP
OpenCharV2
31.3
23,4
40.3
45.4
53.9
42.0
48.8
BLIP2
Llam2
28.6
30.3
44.1
54.6
36.8
46.9
InstructBLIP
Ilam2
31,3
30.3
44.0
45.1
56.7
38.4
48.8
BLIP2
ChatGPT
28.6
35.9
48.0
45.8
56.1
35.2
47.2
InstructBLIP
ChatGT
31.3
35.9
48.0
47.6
58.4
38.1
49.7
Table 4: Question Answering R We group questions based gold reference modality report results breakdown across modalities in table 3. Similarly, group questions answerable by single modality or those requiring multi-modality. The overall R@ 5x3 calculated hased on whether the gold answer found within concatenated 15 answer candidate
score represents the potential maximum perfor- mance, as the answers from different modalities could be harmonized through LLM reasoning.
4.4 Multimodal Open-domain QA Results
Following the rule-based strategy, valid results are processed with the Prompt-Answer Infusion and subsequently reasoned by the LLM. The results are shown in table and table 6.
The results in table indicate that VQA is the most challenging task in MMCoQA due its low recall. In contrast, ChatGPT effectively manages textual question answering for both text and lin- carized tables. In the context of the MultiModalQA dataset, multi-hop reasoning cross-modal under- standing is frequently required, rendering tasks that involve multiple modalities more demanding than those relying on single modality. Our empiri- cal observations shows that some questions can be addressed using references from different modali- ties, achieving an overall Recall @ 5x3 of nearly 50 across both datasets.
The supervised methods on the MMCoQA dataset underperform, largely due to the subpar joint retrieval results as highlighted in table 3. When MAE is provided with gold reference, thereby eliminating the retrieval step, its perfor- mance notably improves witnessing 30.64 increase in Fl score and a 24.58 rise in EM. This implies that the main challenge lies in the retrieval and ranking of references. On the other hand, the Direct QA methods of LLMs effectively handle questions involving textual references, thanks to
Models
Image
Table
Text
Overall
VQA
Textual QA
Direct QA Reusoner
EM
EM
EM
Fl
EM
Supervised
ORConvQAf (Qu al., 2020)
1.87
ManyModiQQA (Talmor et al., 2021) MABÍ (1á et al,, 2022)
1.82
0.96
6.29
3.73
MAE+ + Gold reference)
36.93
28.31
Direct QA
Vicuna-7B (Chiang et al., 2023)
14.1
11.6
12.8
9.0
22.1
17.1
17.8
13.7
OpenChat-v2-w-111 (Wang et al., 2023)
11.3
10.2
19.7
15.9
32.5
25.5
24.1
19.3
Llama2Chat- 13b
17.2
15.0
20.3
14.5
29.9
23.2
24.4
19.0
CharGPT (OpenAI, 2022b)
17.1
14.3
25.9
21.4
45.3
35.6
33.5
26.8
GPT4(OpenAI, 2022c)
22.0
16.3
32.1
26.2
$1.2
44.0
39.2
32.7
Zero-shot MOQAGPT
BLIP2
Vicuna-7B
ChatGPT
23.4
19.7
25.0
20.7
43.6
34.6
34.0
27.5
BLIP2
OpenChur VennChrrV2-13
ChatGPT
25.9
22.4
28.0
23.4
42.8
35.2
35.0
29.2
BLIP2
Llama2Chat-133
ChatGPT
23,9
21.1
35.8
30.3
45.3
36.9
37.7
31.4
BI.IP2
CharGPT
ChaiGPT
24.8
21.1
38.9
33.1
47.5
37.6
39.7
32.4
InstructBI.IP
ChaiGPT
CharGPT
28.5
25.9
36.4
29.7
46.4
37.6
39.5
32.7
InstrucIBLIP
Llama2Chat-
Llama2Chat-1 13b
22.9
18.4
31.5
26.4
41.5
34.5
34.2
27.8
InstructBLIPP
ChatGP
JJamaZChat-133
23.5
18.9
32.8
27.2
42.9
35.4
35.6
28.9
BLIP2
CharGPT
GPT4
28.3
22.4
41.3
35.9
52.8
45.0
43.9
37.1
InstructBLIP
ChaiGPT
GPT4
27.6
23.1
42
35.2
53.6
46.3
44.2
37.8
Table 5: Results on MMCoQA i represents quoted results. VQA represent models to extract answers from image, Textual QA represents models to extract answer from text and lincarlized table. Direct QA & Reasoner represents model which is used to directly ask for and model to infuse information and reasoning and generate final answer

---

Models
Single Modality
Multi Modality
Overall
VQA
Textual QA
Direct QA & Reasoner
EM
HM
F1
EM
Supervised
MGTi (He and Wang, 2023) SKURÖf (Yang al., 2022)
57.7
52.1
70.2
66.3
56.4
51.3
63.8
59.4
Direct QA
Vicuna-7B(Chian al., 2023)
20.3
17.1
16.4
11.9
18.6
14.9
OpenChat-v2 v2-w-13b (Wang et al., 2023)
25,3
22.0
18.9
15.5
22.5
19.2
Llama2Chat-133
24.6
21.3
17.3
13.0
21.3
17.7
ChatGPT (OpenAl, 2022h)
36.9
29.8
22.3
17.4
30.6
24.5
GPT4 (OpenAl, 2022c)
42.9
36.4
27.2
23,9
36.1
31.0
Zero-shot MOQAGPT
B1.IP2
Vicuma-7B
ChatGPT
41.3
34.4
24.7
20.2
34.2
28.3
B1.IP2
OpenChatV2-13b
ChatGPT
40.9
34.4
24.8
20.6
34.0
28.5
BLIP2
Llama2Chat-130
ChatGPT
43,3
36.8
27.5
23,2
36.5
31.0
BLIP2
ChatGPT
ChatGPT
37.0
22.6
36.6
30.8
InstructBLIP
ChatGPT
ChatGPT
43.5
37.2
27.4
23,4
36.6
31.3
InstructBLIP
Llama2Chat-13b
Llama2Chat-
38.5
33.4
24.6
31.4
28.0
InstructBLIP
ChatGPT
(lama2Chat-131
38.6
33,9
24.5
20.4
32.0
28.9
BLIP2
ChatGPT
GPT4
50.6
44.0
31.4
27.5
42.3
36.9
InstructBI.IP
ChatGPT
GPT4
54.6
49.1
33.8
30.5
45.6
41.1
Table 6: Results on MultiModalQAA represents quoted results.
their expansive knowledge storage. Yet, they falter for queries demanding image references, primarily due to modality limitations. Our zero-shot method outshines the supervised baseline because of perior retrieval, question answering, and answer infusion capabilities. It also clevates the Direct QA approach when grounded in retrieved results, showing to a 6.0 Fl and 5.9 EM boost over ChatGPT and 5.0 FI and 5.1 EM enhancement over GPT4. Overall, our methodology exhibits significant improvement across all tested models. The MultiModalQA dataset provides 1-15 refer- ences for each modality, diminishing the criticality retrieval from an extensive multi-modaa knowl- edge base. Thus, our divide and conquer approach might not realize its utmost potential here. Con- sequently, our zero-shot method trails the super- vised baselines. This is in stark contrast MM- CoQA. where retrieval across modalities is impera- tive. Such foundational differences underscore the varied baseline results between the two datasets. However, given that our approach operates zero-shot fashion, devoid of task-specific annota- tions and specialized tools like question classificrs, MOQAGPT notably betters zero-shot baselines and closes the performance gap with supervised meth- ods.
and 10.1 EM over GPT4. The most significant xapр comes from the Single Modality category, un- derscoring the efficacy of approach for one- hop tasks. We also register improved scores in the Multi Modality category, showcasing the abil- ity of our GPT to amalgamate different modalities. Predictably, GPT4 employed for direct QA and reasoning, exhibits superior gains than ChatGPT across both Single/Multi Modality categories. This aligns with our hypothesis: given the task's em- phasis on cross- moosl reasoning, method leans heavily robust reasoning capabilities to merge in- formation across modalities. Thus, the more adept the reasoner, the higher the performance. Moreover, GPTs capably filter out noise from Vicuna's output, markedly enhancing the performance against the direct QA by Vicuna for both datasets.
In conclusion, it's essential to underscore that real- weal-world situations more closely mirror the MM- CoQA setup, where evidence isn' readily available but requires retrieval from vast repositories. In such scenarios, the strengths and merits of our method shine through, substantially surpassing supervised methods, heralding broader acceptance and use.
4.5 Ablation study
In our pursuit assess the efficiency of the pro- posed rule-based strategy, especially its efficacy noise mitigation, we conduct experiments on MM- CoQA. We utilize InstrucBLIP for VQA, ChatGPT
As illustrated in table in comparison to Di- rect QA, our method boosts the overall metrics by 6.0 Fl and 6.8 EM over ChatGPT, and 9.5 FI

---

Quection:: 1: what the compettionn Eat inttarelag with match agents Ronger??
Question:: what Rangers 2010-11 playrr has a shaved nead?
Questiond what's the team with logo feotuiig a hand holdhng a farmig toroh?
Quectioné which courtry was the first land on ste moon
Gold uafa
Naguu
Gold Arsower:
Gold yark
Gold
Candidates: Noothal Eenspaleeggutl
Candidaree: Salan. maddd hougherl))
Candidater: Nits indana paoes)
Candidatee: (xtm. DrA MS8
Final Answwer: Ecropa lsuque
x
Arsoer ollon
Final Asseeo Indiena pacar
Fixal Answer 100
Quartiand: what are the raayers Rangers 20/0:11 League Teans ofe Yeur?
Quectiontc where war the Stape Whow Oinea?
Queetion:: many coloriare Foods in 1952?
Qumetion::: skoa mary oifieent pattomme Caponiaagon are seai hee?
Gold Anowwer wilen
Gold Answwer: Tororto
Gold Arewae:? 2
Gold Amsave::
Candidates: Jatan magregor staven davl)
Candidates: Baromto ontani,, Tononss
Candidates: 22 2)
Candidates: (4 4altfer plotur unckea!
Firal eaggeego
Figure 4: Cases.
Blue text are Direct GPT answers. Q 1-4 are failure cases, Q 5-8 are successful cases.
Image (FI/EM)
Table (FI/EM)
Text
Overall (FI/EM)
#API
tion 6 suggests a link between retrieval-sourced answers and Direct GPT responses. Question 7 depicts that queries can be solved through retrieval methods or directly querying GPT. Lastly, Question 8 demonstrates that our framework equips LLMs with the capability to address tasks which would typically confound vanilla LLMs. More examples to demonstrate interpretability are described in ap- pendix C.
Method
(FI/EM)
times
MOQAGPT
27.1/23.1
42/35.2
53.6/46.3
44.2/37.8
423
vdo Rulel
26.7/22.5
41.6/34.9
52.4/44.2
43,3/36.2
590
who Rule2
24.6/19.8
38.6/31.4
49.8/43.2
40.8/32.9
423
who Rule3
26.6/21.4
39.4/32.7
50.2/43.9
41.8/33.4
423
Table 7: Ablations on rule-based strategy
for TextualQA, and GPT-4 for Direct QA and Rea- soning. Detailed findings from these experiments are presented in table
5 Conclusion
Rule proves to be essential, leading to a 23% reduction in GPT activations. This obviates the need for reasoning over potentially noisy answers, thereby enhancing response accuracy and curtail- ing inference time. The sensitivity of LLMs to input noise, as underscored in Zhang et al. (2023),
In this study, we introduce the first zero-shot multi- modal open-domain question answering frame- work, MOQAGPT, which enables LLMs per- form the MMOQA task. This framework is flex- ible, accommodating new models and modalities without requiring additional training. It stands out for its trustworthiness, being grounded on reliable retrieval results, and its interpretability, which is ensured by transparent intermediate outcomes. By leveraging LLMs and VLMs. our model surpasses supervised methods in MMCoQA performance and significantly narrows the gap between zero-shot and supervised methods in multi-hop Multimodal QA datasets. Furthermore, our results indicate that models without robust knowledge storage capabil- ities, such as Vicuna, are less suited for this task. We hope that our approach offers some insights and servers general and promising framework for multi-modal open-domain question answering.
reinforces the importance of Rule 2. Excluding this rule introduces detrimental noise during the reasoning stage, adversely affecting the outcomes, as corroborated by ablation studies. Rule 3, which refines response selection by assessing the con- sensus among top references, is further validated through ablation study. Collectively, these find- ings cement the role of our rule-based strategy pivotal, optimized element, rather than just a rudi- mentary heuristic.
4.6
Case Study
Fig presents various instances of model perfor- mance. Questions 1-4 show failures: Question 1 shows the string matching metric failing to pro- cess similar meanings. Question 2 illustrates the model's inability to choose the correct answer from the candidate list. Question 3 and 4 highlight the proposal of incorrect candidates and the existence of a knowledge bias in the model, respectively. Conversely, Questions 5-8 exemplify successes: Question 5 shows hallucination errors being recti- fied via grounded retrieval-baa based answers. Ques-
Limitation
Central to our work is the dependency on Large Language Models (LLMs), particularly the GPT family, which being proprietary, necessitates an API call, incurring both financial and temporal costs to replicate our results. While the datasets used in our studies incurred minimal costs (2S and 5$), larger datasets like WebQA could demand

---

more?. The consistent updates the GPT version imply that results, while not precisely reproducible, should only improve compared to those reported in this paper. Furthermore, we provide results from open-source LLMs, ensuring reproducibility.
References
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2015. Neural module networks. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 39-48.
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. 2016. Learning to compose neural net- works for question answering. In North American Chapter of the Association for Computational Lin- guistics.
Ethics Statement
The proposed method, MOQAGPT, offers substan- tial advances in the field of multi-modal open- domain question answering, an area of growing importance in AI. By leveraging large language models (LLMs) like GPT-4, it fosters their ability to handle tasks in zero-shot manner and provides robust solution for extracting and ranking an- swers from databases encompassing variety of modalities such as images, tables, passages, etc. The impact of this work extends across various sectors. By improving the efficiency and effective- systems, we anticipate
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin John- son, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent Shafey, Yanping Huang, Kathy Hellste,,,,,,en,, Gau- rav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Tay, Kefan Xiao. Yuanzhong Xu, Yujing Zhang, Gustavo Hernandez Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Bradbury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng, Colin Cherry, Christopher A. Choquette-Choo, Aakanksha Chowdhery. Clément Crepy, Shachi Dave, Mostafa Dehghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du Ethan Dyer, Vlad Feinherg, Fangxiaoyu Feng. Vlad Fienber, Markus Freitag, Xavier Gar- cia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur- Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui. Jeremy Hur- witz, Michael Isard, Abe Ittycheriah, Matthew Jagiel- ski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee, Ben- jamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Marcello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra. Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nys- trom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope. Siyuan Qiao. Emily Reif, Bryan Richter, Parker Riley. Alex Castro Ros, Au- rko Roy, Brennan Saeta. Rajkumar Samuel. Renee Shelby, Ambrose Slone, Daniel Smilkov, David R. So. Daniel Sohn, Simon Tokumine, Dasha Valter. Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang. Zirui Wang. Tao Wang, John Wiet- ing, Yuhuai Wu, Kelvin Xu, Yunhan Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng. Ce Zheng. Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. 2023. Palm 2 technical report.
ness of guestion-answert that this research will enhance user interactions in digital environments, streamline the retrieval of information, and significantly contribute to the de- velopment of more intuitive. accessible Al tools. In the education sector, for example, the frame- work can be used create more interactive learn- ing systems, making easier for students to extract accurate and comprehensive information from di- verse learning materials. Additionally, in business and research domains, it could expedite data analy- sis by facilitating the retrieval of relevant data from vast, multi-modal databases.
While the enhancement in performance is no- table, as with all Al technology, this research also presents potential societal risks. There might be an increased reliance on Al for answering ques- tions, potentially reducing critical thinking abilities if over-relied upon. As the proposed method can cope with any modality, misuse of the technology might lead to privacy issues if it's used to retrieve sensitive information from various modalities with- out consent.
Acknowledgements
Gasser Auda and Mohamed Kamel. 1999. Modular neural networks: a survey. International journal of neural systems, 902):122-15)
We are grateful to the Mila IDT team for their tech- nical support with the computational infrastructure. The authors acknowledge the material support of NVIDIA in the form of computational resources. During this project, Aishwarya Agrawal was sup- ported by the Canada CIFAR AI Chair award.
Farooq Azam. 2000. Biologically inspired modular neural networks. Ph.D. thesis, Virginia Polytechnic Institute and State University.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah. Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda
PPiicnn as of June 2023

---

Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler. Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Zhenghao Liu, Chenyan Xiong, Yuanhuiyi Lv, Zhiyuan Liu, and Ge Yu. 2023. Universal vision-language dense retrieval: Learning a unified representation space for multi-modal retrieval.
Pan Lu, Baolin Peng. Hao Cheng. Michel Galley, Kai- Wei Chang, Ying Nian Wu, Song-Chun Zhu, and Jian- feng Gao. 2023, Chameleon: Plug-and-play compo- sitional reasoning with large language models. ArXiv, 09942.
Yingshan Chang, Mridu Narang, Hisami Suzuki, Gui- hong Cao, Jianfeng Gao, and Yonatan Bisk. 2022. Wehqa: Multihop and multimodal qa.
OpenAL. 2022a. Ada: Openai's multi-modal research platform. https://openaa wtt.:/oo///eeee com/ressarch/atee
Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W. Cohen. 2022. Murag: Multi- modal eettreval-augmeente generator for open ques- tion answering over images and text. ArXiv, abs/2210.02928..
OpenAl. 2022b. Chatgpt. https://openai.com research/chatgpt.
OpenAL. 2022c. Gpt-4, https:// httpss///ppaai..cmgggtttttt com/gpt-4. gpt-4.
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong, Hong Wang. and William Yang Wang. 2020. Hy- bridQA: dataset of multi-hop question answering over tabular and textual data. In Findings of the Asso- ciation for Computational Linguistics: EMNLP 2020, pages 1036, Online. Association for Computa- tional Linguistics.
Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit lyyer. 2020. Open-retrieval con- versational question answering. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM.
Wei-Lin Chiang, Zhuohan Li. Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try, Amanda Askell, Pamela Mishknn, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn- ing transferable visual models from natural language supervision.
Tanmay
Gupta and Aniruddha Kembhavi. 2022. Vi- sual programming: Compositional visual reasoning without training.
Timo Schick, Jane Dwivedi-YY Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer. Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. ArXiv, abs/2302.04761.
Darryl Hannan, Akshay Jain, and Mohit Bansal. 2020. Manymodalqa: Modality disambiguation and qa over diverse inputs.
Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming La, and Yueting Zhuang. 2023. Hugging- gpt: Solving ai tasks with chatgpt and its friends in hugging face.
Xuchai He and Xin Eric Wang. 2023. Multimodal graph transformer for multimodal question answer- ing. ArXiv, abs/2305.0058)
Jonathan Herzig, Pawel Krzysztof Nowak, Thomas Müller, Francesco Piccinno. and Julian Martin Eisen- schlos. 2020. Tapas: Weakly supervised table parsing via pre-training. arXiv preprint
Dídac Surís, Sachit Menon, and Carl Vondrick. 2023. Vipergpt: Visual inference via python execution for reasoning.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Han- nanch Hajishirzi, and Jonathan Berant, 2021. Mul- timodalqa: Complex question answering over text, tables and images.
Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao Jian-Yun Nie, and Ji-Rong Wen. 2023. Helma: A large-scale hallucination evaluation benchmark for large lan- guage models. arXiv preprint 233.5:2305.11744
Yongqi Li. Wenjie Li, and Liqiang Nie. 2022. Mmcoqu: Conversational question answering over text, tables, and images. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4220-442 -231.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models.
Tsung- Yi Lin, Michael Maire. Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitncck and Piotr Dol- lár. 2014. Microsoft coco: Common objects con- text.
Guan Wang. Sijie Cheng, Qiying Yu, and Changling Liu. 2023. OpenChat: Advancing Open-source Language Models with Imperfect Data.

---

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian lchter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. 2023. Chain-of-thought prompting elic- its reasoning in large language models.
Chenfei Wu, Sheng-Kai Yin, Weizhen Qi. Xiaodong Wang, Zecheng Tang, and Nan Duan. 2023. Visual chatgpt: Talking, drawing and editing with visual foundation models. ArXiv, 155/2333.04671.
Lee Xiong. Chenyan Xiong. Li, Kwok-Fung Tang. Jialin Liu, Paul N Bennett, Junaid Ahmed, and Amold Overwijk. 2020. Approximate nearest neigh- negative contrastive learning for dense text re- trieval. In International Conference on Learning Representations.
Qian Yang. Qian Chen, Wen Wang, Baotian Hu, and Min Zhang. 2022. Enhancing multi-modal and multi- hop question answering via structured knowledge and unified retrieval- generation. ArXiv, abs/2212.08632.
Peter Young, Alice Lai, Micah Hodosh, and Julia Hock- enmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic in- ference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Zhen Zhang, Guanhua Zhang, Bairu Hou, Wenqi Fan. Qing Li, Sijia Liu, Yang Zhang, and Shiyu Chang. 2023. Certified robustness for large lan- guage models with self-denoising. arXiv preprint ar7iv.230..07171

---

Nout
swer recall due CoT does not aid in final answer extraction fig. 5. A possible reason that CoT extracts larger quantity of information from the reference material, both useful and irrelevant. It even attempts to provide answers when sufficient information isn' 't available, leading not only to in- clude correct answers but also incorrect ones in the candidate pool. For instance, in MMCoQA, the average number of valid answer candidates with CoT is 3, compared 2.5 without it. This addition of noise could confuse GPT4, impairing its ability to make the correct decision.
Figure 5: QA with/without CoT.. Same metrics as ta-
ble VQA is InstructBLIP.
A Does chain-of-thought help?
CoT reasoning represents an emerging capability within LLMs. Given that we employ LLMs as our Textual QA and Reasoner, is pertinent to examine if CoT aids in our setup. To address this, we im- plement a straightforward strategy, adopting spell prompt that reads: /reference) (question) Let's think step by step. This results in the model gener- ating a step-by-step reasoning response, which con- templates the reference in relation to the question, and evaluates if sufficient information available for a response. Subsequently, we prompt GPT to extract an answer, considering the question, reason- ing process, and reference with the prompt: Rea- coning:(reasonie) Question:[queet [question] Give very short answer, in one or two words. Upon conducting this process, we observe the following findings:
B Implementation details
As method employs a zero-shot approach in- volving only inference, all experiments were con- ducted on a single A100 GPU. We employed CLIP: VTT-B/32 for image retrieval, text-embeddingg ada-002 for table retrieval, and ANCE-roberta for text retrieval. We apply BLJP2-FlanT5xx and InstructBLIP-vicuna- for Visual Question An- swering (VQA), gpt-3.5-turbo, OpenChat-v2-iy 13b_Llama2Chut-333 and vicuna-7B for textual QA.
C Detailed Example
Our methodology is detailed in table table 10 and table 11, showcasing the retrieval results, question- answering process, strategy outputs, and final an- swer fusion. For these examples, BLIP2 serves the VQA model, ChatGPT as the textual QA model, and GPT4 as the reasoner and direct QA model. The retrieval results across all modalities are ratio- nal, despite several 'Unknown' instances, which are filtered out through the strategy. While the fi- nal answers are expected to be correct, they do not meet the current exact match criteria. The veracity of the results is established by grounded answer sources. For instance, in table 10, while GPT's answer is ungrounded and incorrect, our method- ology provides accurate information enabling the LLM to select the correct choice.
Firstly, CoT proves beneficial for retrieval-based question answering as demonstrated in fig. 5. The technique enables LLMs to better extract potential answers from references, significantly improving recall for table, text, and overall data. However, for Direct QA, all metrics except GPT4's Fl score decrease. This is because CoT focuses reason- ing which isn't necessary for answering general knowledge questions. We've noticed that with CoT, GPT4 tends to generate longer results, thus improv- ing its FI score.
Reasoner CoT
Recall @ 15
F1
EM
ChatGPT
X
47.6
39.5
32.7
ChatGPT
V
55.3
39.6
32.9
GPT4
47.6
44.2
37.8
GPT4
55.3
44
36.8
Table 8: CoT Ablation Results with Different Rea- soners VQA model is InstructBIIIP and Textual QA ChatGPT
Secondly, it's unexpected that the increased an-

---

What is the competition of Gtu irttarfelag with match against Rangers? Gold reference modality: table; Answer: uefa champion leagu
Question:
# Image Retrieval
VUPPERTALER O
D
C
SPORTVERET
# Text Retrieval
1. Knatspyynufllagid Ægir is Icelandic sports club from the town of Porlákshöfn, mainly known for its football team. The club has football team playing in the fifth tier of Icelandic football.
2. Torfnesvöllur, known as Olfsvöllurinn for sponsorship reasons, is a football stadium in Isafjördur, Iceland and the home of Vestri and Knattspyrnufélagi Hördur. It broke ground in 1963...
3. Tvillingdebyt (. "The Twin Derby") is a football fixture in Stockholm, Sweden, between cross-town rivals AIK and Djurgärdens IF. Both clubs were founded in Stockhoin in 1891, just three weeks
4. The Asturian derby (, or "Derbi astur"), is the name given to any association football match contested between Sporting de Gijón and Real Oviedo, the two biggest clubs in Asturias. The rivalry ...
5. Knatspyynuflaaio Hördur was founded on 27 May 1919 as a football club with Pórhallur Leósson being first chairman. Its first official game was against Fótboltafélag İsafjardar on 17 June 1921. # Table Retrieval
1. European Cup UEFA Champions League European Cup UEFA Champions Leugue European
Cup / UEFA Champions League European Cup / UEFA Champions
2. 2005-06 UEFA Cup 10 FC Nistru Otaci 1-2 1-3 2-5 2007-08 UEFA Champions League 1Q NK Dinamo Zagreb 1-1 1-3 (aet) 2-4 2008-008 09 UEFA Cup
3. 1970-71 UEFA Cup Winners' Cup 1 Partizani 3-2 2-1 5-3 Real Madrid 0-2 1-0 1-2 1971-72 European Cup 1 Benfica 1-3 0-4 1-7 1972-73..
4. 1963-64 European Cup Preliminary round Borussia Dortmund 2-4 1-3 3-7 1964-65 European
Cup Prelininary round Reipas Lahti
1-24-2..
5. 1968-69 69 European Cup Winners' Cup First Round FC Barcelona 0-1 0-4 1971-72 UEFA Cup
First Round Legia Warszawa 1-30-0 1-3 1993-94
# Question Answering Prompt
You are performing extractive question answering. Given the document: (reference). extract short answer to the question: question from the document. If insufficient information is available to answer the question, respond with 'Unknown'. The answer should be one or two words long.
# Image Answer
wuppertaler, woman football, league cup, league cup, league cup
# Text Answer
Unknown., Unknown., Unknown., Unknown., Unknown.
# Tabble Answer Unknown., Unknown., Unknown. Unknown., Unknown.
Chatgpt Answer
europa leagu
Valid Answer Candidates(after strategy)
league cup, wuppertal, eurrpa leagu
Answer Fusion
Given question (0), please select the best answer from the following candidates: /Candidates) # Final Answer
europa leagu
Table 9: Examplel, detailed results of MogaGPT solve the task, note that there are repeated images exist in the dataset

---

Question: how many songs were written by john lennon and paul mecartney Gold reference modality: text; Answer: 180
# Image Retrieval
JOHN LENNON IMAGINE
John
BomE Leninnn Iwyy
# Text Retrieval
1. Lennon McCartney was the songwriting partnership between English musicians John Lennon (9
October 194000a0 December 1980) and Paul McCartney.
2. Lennon McCarmey was the songwriting partnership between English musicians John Lennon (9 October 194000a0 December 1980) and Paul McCartney...
3. Lennon McCartey was the songwriting partnership between English musicians John Lennon and Paul McCartney of the Beatles, the partnership published approximately 180 jointly credited songs
4. Paul McCariney is English musician who has recorded hundreds of songs over the course of his over 60-year career, As member of the Beatles, he formed songwriting partnership with bandmate John
5. Unlike many songwriting partnerships that comprise separate lyricist and composer, such as Jerry Leiber and Mike Stoller, Rodgers and Hammerstein, # Table Retrieval
1. 75px Dylan May 24, 1941 present Mixed-Up Up Confusion (1962), performed by himself 2 75px McCartney June 18, 1942 present Love Me Do/P.S. Love You
2. 1974 McGear Mike McGear 1980 The Reluctant Dog Steve Holley 1981 Somewhere in England All
Those Years Ago George Harrison 1982 Tug of War
3. 196? Words and Music by Paul Williams Big Seven Music Corp. 1970 Someday Man Reprise...
4. I If You Be My Baby Peter Green/Clifford Adams Fleetwood Mac Mr. Wonderful (1968) 6:38 Long
Grey Mare Peter Green Fleetwood Mac ...
5. Disc (23971): Disc 1 (23971): Disc (23971): Disc I (23971): Disc (23971): A. Love You Truly Carrie Jacobs Bond April 18, 1945 John Scott Trotter and His Orchestra 2:56 B. Just,
# Question Answering Prompt You are performing extractive question answering. Given the document: (reference). extract short answer to the question: question from the document. If insufficient information is available to answer the question, respond with 'Unknown', " The answer should be one or two words long.
# Image Answer
more than one hundred, just like starting over, the john lennon story, imagine, more than one hundred # Text Answer
Approximately 180., Approximately 180., Approximately 180., Unknown., Unknown.
# Tabble Answer
Two songs were written., Unknown., Unknown, Unknown. Unknown.
Chatgpt Answer
Over 150
Valid Answer Candidates(after strategy)
more than 1 hundr.2 approxin 180 jointli credit song.over 150
Answer Fusion
Given question (Q), please select the best answer from the following candidates: (Candidates)
# Final Answer
approxim 180
Table 10: Example2, detailed results of MoqaGPT solve the task

---

What kind of coaster is shown in this photo collage of Copenhagen? Gold reference modality: image; Answer: roller
Question:
# Image Retrieval
# Text Retrieval
1. The ride's is station is located on the midway directly across from Top Thrill Dragster and the first coaster to have inversions featuring a walkway underneath. The ride consists of
2. Kong is a steel Suspended Looping Coaster, made by Vekoma, located at Six Flags Discovery Kingdom in Vallejo, California...
3. Diavlo is a steel roller coaster at Himeji Central Park in Japan which a clone of Batman the Ride.
It is one of the first...
4. The first inversion in roller coaster history was part of the Centrifugal Railway of Paris, France, built in 1848. It consisted of a 43-foot (13-meter) sloping track...
5. Colossus is steel roller coaster at Thorpe Park in Surrey, England, and the park's 's first major attraction. It was built by Swiss manufacturers Intamin and designed...
# Table Retrieval
1. Wolverine Wildcat Michigan's Adventure United States 1988 Raging Wolf Bobs Geauga Lake United States 1988 Timber Wolf Worlds of Fun United States 1989 Hercules Dorney Park United States 1989 2. The Bat 1987 Vekoma A Vekoma Boomerang roller coaster. It was the seventh roller coaster added to the park. The Bats train was originally from the parks Dragon Fire coaster. During the 2008 season
3. 1985 Congo Carousel Robert Tidman Classic gallopers ride, operated previously at Happy Hour Amusement Park, Colwyn Bay 1986 Jungle Swings A classic chair-o-plane ride 1986 Jungle Cat ...
4. All Saints' Church Church of Denmark 1932 150px Drag Church Church of Denmark 1885 150px Hans Tausen's 's Church Church of Denmark 1924 150px Hdevang Church Church of Denmark 1885 150px..
5. Ny Ellebjerg F, A, E 16 November 2006 16 November 2006 transfer to K00f8ge radial Gl. K00f8ge Landevej 2014 8 January 2005 8 January 2005 Temporary terminus;..
Question Answering Prompt
You are performing extractive question answering. Given the document: (reference]. extract a short answer to the question: question from the document. If insufficient information is available to answer the question, respond with 'Unknown'. The answer should be one or two words long.
# Image Answer
roller coaster, roller coaster, roller coaster, wooden coaster, heart,
# Text Answer
Unknown. Unknown., Unknown., Unknown., Unknown.
Tabble Answer Unknown., Unknown., Unknown. Unknown., Unknown.
# Chatgpt Answer
Tivoli Gardens
Valid Answer Candidates(after strategy)
roller coaster, tivoli garden
# Answer Fusion
Given question (Q), please select the best answer from the following candidates: (Candidates)
# Final Answer
roller
Table 1I: Example3, detailed results of MoqaGPT solve the task


Reviews:
[{"review_id": "cZgkpuAW71", "reviewer": "Reviewer_fDtg", "summary": "", "questions": "How do they measure trustworthiness (lines 129-130)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers' time and effort in examining our work and providing valuable feedback.\n\n> The results in Tables 5 and 6 are not fairly compared with other mono-modal retrieval+generation pipelines. For example, adding a monomodal approach (e.g., textual evidence retrieval + ChatGPT) could benefit the comparison and strengthen or weaken the results.\n> \n\nThank you for bringing this to our attention. Current baselines are based on all related works. Given that our task focuses on multimodal question answering, a central challenge is determining from which modality the answer should be derived. Notably, the sources of answers—whether from images, tables, or passages—are evenly distributed. In Tables 5 and 6, we report results according to the actual modality source. \n\nRelying solely on a mono-modal retrieval, such as using textual evidence retrieval combined with ChatGPT, would likely lead to subpar performance when answers necessitate image or table sources. This is why such approaches are not represented in Tables 5 and 6. However, to address your valid concerns comprehensively, we have compiled additional results:\n\n| MMCOQA | VQA model | Textual QA model | Direct QA | Reasoner | Image(F1/EM) | Table(F1/EM) | Text(F1/EM) | Overall(F1/EM) |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| visual evidence+chatgpt | BLIP2  | None | None| ChatGPT | 17.9/12.5 | 2.2/1.2 | 2.6/0.8 | 10.6/7.2 |\n| tabular evidence+chatgpt | None | ChatGPT | None | ChatGPT | 2.3/0.7 | 25.6/22.1 | 12.6/10.5 | 14.3/8.8 |\n| textual evidence+chatgpt | None | ChatGPT | None | ChatGPT | 1.8/0.9 | 14.6/11.1 | 30.6/26.9 | 15.4/10.9 |\n| MoqaGPT | BLIP2 | ChatGPT | ChatGPT | ChatGPT | 24.8/21.1 | 38.9/33.1 | 47.5/37.6 | 39.7/32.4 |\n\n| MultimodalQA | VQA model | Textual QA model | Direct QA | Reasoner | single modality(F1/EM) | multi modality (F1/EM) | Overall(F1/EM) |\n| --- | --- | --- | --- | --- | --- | --- | --- |\n| visual evidence+chatgpt | BLIP2 | None| None | ChatGPT | 15.6/10.6 | 10.6/6.4 | 11.4/6.4 |\n| tabular evidence+chatgpt |None | ChatGPT | None | ChatGPT | 22.6/16.7 | 12.4/7.0 | 14.6/8.9 |\n| textual evidence+chatgpt | None | ChatGPT | None | ChatGPT | 24.2/17.4 | 15.3/10.2 | 15.2/10.0 |\n| MoqaGPT | BLIP2 | ChatGPT | ChatGPT | ChatGPT | 43.9/37.0 | 26.9/22.6 | 36.6/30.8 |\n\nAs demonstrated, the monomodal approach falls short in this task, we believe this further experiments strengthen our methods, leading to increased soundness. We also incoporate other opensource LLM such as Llama2, please refer to the new  result table in our repsonse to Reviewer LC2A.\n\n> The rule-based strategy sounds heuristic or sub-optimal\n> \n\nThank you for your feedback on the rule-based strategy. We devised this strategy based on comprehensive observations of LLM behaviors and our prior knowledge, as detailed in lines 318-332. \n\n- In our experiments, when a GPT-generated answer matches any of the retrieved responses, it's typically correct. As supported by the ablation study shown below, **Rule1 reduces unnecessary LLM activations by 23%**, eliminates the need for reasoning over potentially noisy multiple answer choices, and consequently, enhances the accuracy of the concluding response. Therefore, **Rule 1 is not only about saving inference time but also about enhancing the performance.**\n    \n    \n    | rule 1 | VQA model | Textual QA model | Direct QA&Reasoner | Image(F1/EM) | Table(F1/EM) | Text(F1/EM) | Overall(F1/EM) | API calling times |\n    | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n    | ✔️ | InstructBLIP | ChatGPT | GPT4 | 27.1/23.1 | 42/35.2 | 53.6/46.3 | 44.2/37.8 | 423 |\n    | ❌ | InstructBLIP | ChatGPT | GPT4 | 26.7/22.5 | 41.6/34.9 | 52.4/44.2 | 43.3/36.2 | 590 |\n- As highlighted in the paper \"Certified Robustness for Large Language Models with Self-Denoising\", current LLMs are highly sensitive to input noise. Without this rule 2, an excessive amount of noise (’sorry’, ‘unknown’, ’i’m a language model…’ etc.) would permeate the final reasoning stage, leading to subpar outcomes. We've conducted ablation studies both with and without this rule 2, and the diminished performance in its absence underscores its pivotal role in the system.\n    \n    \n    | rule 2 | VQA model | Textual QA model | Direct QA&Reasoner | Image(F1/EM) | Table(F1/EM) | Text(F1/EM) | Overall(F1/EM) |\n    | --- | --- | --- | --- | --- | --- | --- | --- |\n    | ✔️ | InstructBLIP | ChatGPT | GPT4 | 27.1/23.1 | 42/35.2 | 53.6/46.3 | 44.2/37.8 |\n    | ❌ | InstructBLIP | ChatGPT | GPT4 | 24.6/19.8 | 38.6/31.4 | 49.8/43.2 | 40.8/32.9 |\n- Rule 3 plays a crucial role in **enhancing the probability of choosing the most precise response by evaluating agreement among several top references**. Here's an ablation study highlighting the impact of including or excluding Rule 3:\n    \n    \n    | rule 3 | VQA model | Textual QA model | Direct QA&Reasoner | Image(F1/EM) | Table(F1/EM) | Text(F1/EM) | Overall(F1/EM) |\n    | --- | --- | --- | --- | --- | --- | --- | --- |\n    | ✔️ | InstructBLIP | ChatGPT | GPT4 | 27.1/23.1 | 42/35.2 | 53.6/46.3 | 44.2/37.8 |\n    | ❌ | InstructBLIP | ChatGPT | GPT4 | 26.6/21.4 | 39.4/32.7 | 50.2/43.9 | 41.8/33.4 |\n\nTherefore, **we believe rule-based strategy is not merely heuristic but a crucial, optimized component**. \n\n> How do they measure trustworthiness (lines 129-130)?\n> \n\nThank you for inquiring about our measure of trustworthiness mentioned in lines 129-130. We evaluate trustworthiness based on our model's performance across two benchmarks including MMCoQA and MultimodalQA. The assertion that our approach results in \"less hallucination and thus greater trustworthiness\" is substantiated by the results presented in Table 4, which shows high answer recall, and in Tables 5 and 6, where we achieve commendable F1/EM scores. A high recall suggests that the correct answer is rooted in our retrieved evidence, and such grounding effectively reduces hallucination. Additionally, Exact Match doesn't favor hallucinated answers, making it a reliable metric to gauge the trustworthiness of a question-answering framework. \n\n### General response to Reviewer ****fDtg****\n\nWe trust that our additional results and expanded explanations address your concerns, further **underscoring the soundness of our work**. We have conducted extensive evaluations on each component of our pipeline, demonstrating the **indispensable nature of each module** we've incorporated. Given the promising results we've achieved on this task, we hope the reviewer would consider increasing the soundness score of our paper", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Their approach addresses an interesting problem, presenting an interesting solution to improve the performance of a QA system\n- The results show promising performance when compared with the presented baselines.\n- The paper is generally well-written and easy to follow.", "reasons_to_reject": "- Despite the promising results, the paper is missing some important baselines. The results in Tables 5 and 6 are not fairly compared with other mono-modal retrieval+generation pipelines. For example, adding a monomodal approach (e.g., textual evidence retrieval + ChatGPT) could benefit the comparison and strengthen or weaken the results.\n- The rule-based strategy sounds heuristic or sub-optimal.", "questions_to_authors": "How do they measure trustworthiness (lines 129-130)?", "paper_topic_and_main_contributions": "This paper addresses the multi-modal open-domain QA task. Specifically, they propose a novel framework employing a divide-and-conquer strategy to provide a correct final answer by fusing multi-modal information from multiple sources and modalities. They called this multi-modal approach MoqaGPT.\n\nSpecifically, they designed a pipeline based on two sequential steps. First, a multi-modal QA extractor extracts the answers in a multi-modal way (for example, considering images and text). Second, an LLM considers the retrieved candidates and generates the most plausible answer.\n\nThey evaluated their approach on two datasets (MMCoQA and MultiModalQA), comparing it with other supervised and zero-shot approaches. The results highlight that their technique looks promising and can outperform most of the presented baselines.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Auto Aggressive line 320"}, "extra_text": {}}, {"review_id": "QLek1ufUT2", "reviewer": "Reviewer_PCcG", "summary": "", "questions": "None", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are genuinely grateful for the reviewers' thorough examination of our work and the invaluable feedback provided. Your positive remarks on the clarity of our writing and the empirical effectiveness of our methods are deeply encouraging. Recognizing our approach as 'simple yet effective' is truly the highest compliment for our research.\n\n> Although the approach performs well, it is still useful to see the error cases in experiments. Since LLMs may generate unstable results, it is necessary to add error case categories.\n> \n\nThank you for highlighting the importance of showcasing error cases. Indeed, the potential instability in LLMs' outputs is a valid concern. To address this, we have detailed several observed error cases within our framework in Figure 3 and lines 556-562. These include:\n\n1. Issues with the string matching metric, where it fails to process semantically similar meanings.\n2. The model's occasional inability to select the correct answer from the candidate list.\n3. As illustrated in Questions 3 and 4, the introduction of incorrect candidates and the model's inherent knowledge bias, respectively.\n\nWe agree that understanding these shortcomings is pivotal to the holistic evaluation and future enhancement of our approach.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.This paper is well-written and easy to follow. \n\n2.The experimental results prove the effectiveness of the proposed method.\n\n3.The designed approach is simple and effective and can be adopted in real scenarios easily.", "reasons_to_reject": "Although the approach performs well, it is still useful to see the error cases in experiments. Since LLMs may generate unstable results, it is necessary to add error case categories.", "questions_to_authors": "None", "paper_topic_and_main_contributions": "This paper focuses on the multi-modal open-domain question answering task, which is also challenging for LLMs like GPT4. In order to tackle this challenge, the authors propose MoqaGTP,  which is a simple and effective framework. It employs a divide-and-conquer method to first retrieveal single-modality resources and combine them together to answer the questions. The experimental results demonstrate the effectiveness of the proposed method and more experimental analysis are performed to support the approach. Overall, this paper is easy to read and follow, and the approach is simple and effective. \n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jghcsvfAxx", "reviewer": "Reviewer_LC2A", "summary": "", "questions": "Why are performances on two datasets so differently? On MMCoQA, the zero-shot retrieval greatly outperforms the previous SOTA MAE. While on MultiModalQA, the zero-shot retrieval perform inferier than previous SOTA.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewers' time and effort in examining our work and providing valuable feedback.\n\n> The similar pipeline appears in many subtask. Merely conducting experiment over different combination of LLM/VLM may be lack of insight.\n> \n\nThank you for noting the recurring pipeline pattern across multiple subtasks. It's accurate that several subtasks employ a similar retrieval-based question answering structure, but we **believe this pipeline best addresses the task** at hand, hence its usage. Our contribution is not a new pipeline but the unique perspective lies in harnessing the capabilities of LLM to amalgamate data from disparate sources and subsequently reason over it. To the best of our knowledge, **we are the first to achieve results for this task using a zero-shot method**. Integrating these pipelines within the framework of LLMs and VLMs, especially for the MMOQA task, introduces distinct challenges, like noise filtration and diverse source information fusion (L077-089). Here are the key insights our approach brings to the table:\n\n1. **Human-like Multimodal Processing:** At the heart of our approach is an emulation of human behavior when faced with multimodal open-domain questions. Humans naturally seek information from diverse sources to compose a coherent answer. Admittedly, even this seemingly intuitive framework poses significant challenges to prior supervised methods, due to:\n    - The challenge lies in accurately identifying the correct modality for sourcing answers, given the inherent ambiguity of this task where a single question can be addressed using references from multiple modalities.\n    - The complexity of merging information from diverse sources, with supervised methods depending on ranking – a process difficult to optimize due to question ambiguity and constraints in training data.\n    - A notable shortage of specifically labeled datasets for this task, including questions, references and answers, given the high costs of dataset collection.\n2. **Unsupervised & Zero-shot Nature and impressive results:** While our method may appear intuitive, its **innovation lies in its unsupervised, zero-shot approach to tackling the task**. As per our understanding, ours is the **first framework to address multimodal open-domain questions without relying on specific annotations**, harnessing the remarkable generalization capabilities of the LLM. Our method outperforms the previous supervised state-of-the-art on the real-life scenario setting of MMCoQA, underscoring the promising potential of our framework.\n3. **Significance of Each Module:** Our framework's design wasn't straightforward. **Every module is integral to the overall performance** (see our additional results table to Reviewer fDtg and new results table below). For instance, the absence of rule-based filtering would plummet our scores from F1 44.2 to 40.8 and EM 37.8 to 32.9.\n\nIn conclusion, while simplicity and intuitiveness can sometimes be mistaken for lack of novelty, they can be strengths in themselves.  As suggested in ACL reviewer guidelines “**The goal is to solve the problem, not to solve it in a complex way. Simpler solutions are in fact preferable, as they are less brittle and easier to deploy in real-world settings.**” \n\nThe takeaway from our work is threefold:\n\n- Deploying powerful pre-existing models in a zero-shot manner is not only viable but also potent.\n- It often leads to performance enhancements and broadened applicability.\n- Reasoning over multiple answers from different sources are the most important part for MMOQA task (see ********our new results with Llama2 shown below)\n\nWe hope this sheds light on our method's insight.\n\n> Incomplete ablation study in Table 5 and Table 6. Only ChatGPT family is selected as direct QA & Reasoner.\n> \n\nWe utilized the latest open-source model, Llama2Chat-13b, as a direct QA & Reasoner, as detailed below. As evident, within our framework, **Llama2 consistently delivers commendable outcomes**. Larger models such as GPT family exhibit superior performance in direct QA tasks, given their knowledge-intensive nature. Furthermore, the table illustrates that the **reasoner plays a more crucial role than the Textual Question Answering model**. This underscores the primary challenge: effectively amalgamating information from diverse sources. \n\n### MMCoQA\n\n| VQA model | Textual QA model | Direct QA&Reasoner | Image(F1/EM) | Table(F1/EM) | Text(F1/EM) | Overall(F1/EM) |\n| --- | --- | --- | --- | --- | --- | --- |\n|  |  | Llama2Chat-13b | 17.2/15.0 | 20.3/14.5 | 29.9/23.2 | 24.4/19.0 |\n|  |  | ChatGPT | 17.1/14.3 | 25.9/21.4 | 45.3/35.6 | 33.5/26.8 |\n|  |  | GPT4 | 22.0/16.3 | 32.1/26.2 | 51.2/44.0 | 39.2/32.7 |\n| InstructBLIP | Llama2Chat-13b | Llama2Chat-13b | 22.9/18.4 | 31.5/26.4 | 41.5/34.5 | 34.2/27.8 |\n| InstructBLIP | ChatGPT | Llama2Chat-13b | 23.5/18.9 | 32.8/27.2 | 42.9/35.4 | 35.6/28.9 |\n| InstructBLIP | Llama2Chat-13b | ChatGPT | 27.9/25.1 | 35.8/30.3 | 45.3/36.9 | 38.7/31.4 |\n| InstructBLIP | ChatGPT | ChatGPT | 28.5/25.9 | 36.4/29.7 | 46.4/37.6 | 39.5/32.7 |\n| InstructBLIP | ChatGPT | GPT4 | 27.6/23.1 | 42.0/35.2 | 53.6/46.3 | 44.2/37.8 |\n\n### MultimodalQA\n\n| VQA model | Textual QA model | Direct QA&Reasoner | single modality(F1/EM) | multi modality (F1/EM) | Overall(F1/EM) |\n| --- | --- | --- | --- | --- | --- |\n|  |  | Llama2Chat-13b | 25.3/22.0 | 18.9/15.5 | 22.5/19.2 |\n|  |  | ChatGPT | 36.9/29.8 | 22.3/17.4 | 30.6/24.5 |\n|  |  | GPT4 | 42.9/36.4 | 27.2/23.9 | 36.1/31.0 |\n| InstructBLIP | Llama2Chat-13b | Llama2Chat-13b | 38.5/33.4 | 24.6/19.8 | 31.4/28.0 |\n| InstructBLIP | ChatGPT | Llama2Chat-13b | 38.6/33.9 | 24.5/20.4 | 32.0/28.9 |\n| InstructBLIP | Llama2Chat-13b | ChatGPT | 43.3/36.8 | 27.5/23.2 | 36.5/31.0 |\n| InstructBLIP | ChatGPT | ChatGPT | 43.5/37.2 | 27.4/23.4 | 36.6/31.3 |\n| InstructBLIP | ChatGPT | GPT4 | 54.6/49.1 | 33.8/30.5 | 45.6/41.1 |\n\n> Why are performances on two datasets so differently? On MMCoQA, the zero-shot retrieval greatly outperforms the previous SOTA MAE. While on MultiModalQA, the zero-shot retrieval perform inferier than previous SOTA.\n> \n\nThank you for bringing up the performance discrepancies across the two datasets. It's worth mentioning that, **across various question-answering tasks (VQA and textual QA), supervised methods typically have the edge over zero-shot approaches**. The discrepancies in performance can be traced back to the inherent task settings of MMCoQA and MultiModalQA, despite both belonging to the domain of multimodal open-domain question answering. \n\n1. **MMCoQA Task Setting:** Within this dataset, the retrieval task involves sourcing 5 images from a pool of 57,058, 5 tables out of 10,042, and 5 passages from a large pool of 218,285. The prior state-of-the-art method, MAE, deploys a multimodal retrieval approach, training modalities to map into a shared embedding space. Given the dataset's size limitations and the plethora of modalities involved (query, table, image, passage), **this space is notably challenging to perfect**. The struggles associated with this method are evident in Table 3, showcasing a recall@2000 of merely 63.4%. Consequently, **the subsequent answer extraction suffers due to suboptimal retrieval outcomes, thus impacting overall performance**. Rather than using a multi-modal embedding space for retrieval, our framework employs a 'divide and conquer' strategy for each modality and thus **recall of evidence is significantly higher than previous SOTA MAE**. By harnessing the powerful reasoning abilities of the LLM to assimilate information  and deliver the final decision, our method **outperforms its supervised counterparts**.\n2. **MultiModalQA Task Setting:** Contrarily, each question in MultiModalQA provides a **maximum of 15 candidate references**. Notably, **baseline methods sidestep the retrieval process entirely**, leveraging the 15 available evidences for answer derivation. This is why their performance noticeably exceeds that of supervised methods on MMCoQA (where retrieval is a must). As articulated in lines 515-531, **the primary advantage of our \"divide and conquer\" strategy does not come into play** under these conditions, leading our zero-shot method to fall short of supervised counterparts (which is quiet common for question answering tasks). \n    \n    Yet,  **our work is first zero-shot method for this task**, we significantly narrow the performance disparity between supervised and naive zero-shot approaches (direct question answering). It's pivotal to highlight that **our method operates without relying on training annotations**, rendering it genuinely zero-shot—a distinct advantage in its own right.\n    \n\nLastly, it's crucial to emphasize that **real-life scenarios resonate more closely with the MMCoQA setup**, where **candidate evidences aren't hand-delivered** but necessitate retrieval from expansive databases. In such practical contexts, our method's **applicability and advantages become evident, significantly outperforming supervised approaches, paving the way for easy adoption**.\n\n### General response to ****Reviewer LC2A****\n\nWe believe that our additional explanations and results address your concerns, reinforcing the value of our work. We aim to demonstrate that while a single model might not overcome all challenges, a thoughtfully designed ensemble, with an LLM at its core, can potentially tackle many of these challenges in a zero-shot manner. \n\nIt's invigorating for the field to recognize that an LLM can serve roles beyond mere generation, acting as a 'reasoning brain' across varied task settings. Our framework not only delivers commendable results but also offers flexibility to incorporate additional modalities and enhance modules without further training. Hence, we are confident in its **potential as a general solution for multimodal open-domain question answering**. This aspect of our work is what truly excites us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Performance over Retrieval and Question Answering is impressive.\n2. Overall complete ablation study (including multiple combination of LLM and VLM, CoT e.t.c)", "reasons_to_reject": "1. The similar pipeline appears in many subtask. Merely conducting experiment over different combination of LLM/VLM may be lack of insight.\n2. Incomplete ablation study in Table 5 and Table 6. Only ChatGPT family is selected as direct QA & Reasoner.", "questions_to_authors": "Why are performances on two datasets so differently? On MMCoQA, the zero-shot retrieval greatly outperforms the previous SOTA MAE. While on MultiModalQA, the zero-shot retrieval perform inferier than previous SOTA.", "paper_topic_and_main_contributions": "1. The paper proposes a pipeline to integrate LLM into zero-shot Multi-modal Open-domain Question Answering (MMOQA).\n2. Extensive experiments involving multiple combination of LLM and VLM.\n3. Impressive performance improvement over Retrieval and Questions Answering.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for zero-shot multimodal QA, MoqaGPT, which extracts answers in each modality and then later fuses answers using LLMs. The proposed method shows strong performance. While the novelty might be limited, overall the proposed method is well-motivated, sound, and strong. I recommend this paper for acceptance. 
On the other hand, in the current draft, the main ablations of different models are mostly limited to GPT-4 and ChatGPT, which may introduce issues of data contamination or inability to reproduce the results. I strongly recommend authors include the ablation results with more open-sourced models (for reasoners) as shared in https://openreview.net/forum?id=wrBIS6FOfV&noteId=bPutO2Ja6P in the final version as well. While I understand the scaling of models strongly affects the final performance, due to the nature of such closed and proprietary LLM APIs, it is important to include the results of those open-sourced, reproducible LLM results in the paper.
Decision: Accept-Findings