Unifying Text, Tables, and Images for Multimodal Question Answering
Authors: Haohao Luo
Ying Shen
Yang Deng
Venue: EMNLP 2023 Findings (2023)
Keywords: Multimodal Question Answering
Modality Unification
Image Caption

Abstract: Multimodal question answering (MMQA), which aims to derive the answer from multiple knowledge modalities (e.g., text, tables, and images), has received increasing attention due to its board applications. Current approaches to MMQA often rely on single-modal or bi-modal QA models, which limits their ability to effectively integrate information across all modalities and leverage the power of pre-trained language models. To address these limitations, we propose a novel framework called UniMMQA, which unifies three different input modalities into a text-to-text format by employing position-enhanced table linearization and diversified image captioning techniques. Additionally, we enhance cross-modal reasoning by incorporating a multimodal rationale generator, which produces textual descriptions of cross-modal relations for adaptation into the text-to-text generation process. Experimental results on three MMQA benchmark datasets show the superiority of UniMMQA in both supervised and unsupervised settings.

Unifying Text, Tables, and Images for Multimodal Question Answering
Haohao Luo', Ying Shen', Yang Deng**
School of Intelligent Systems Engineering, Sun Yat-sen University National University of Singapore
luohh5email2, sysu. edu. cn, sheny76email. sysuu. edu. cn, ydengênus.edus edu. sg
Abstract
Text-and-lmegge
Text-and-Table
VQA
MMQA
MMQA
TableQA
Multimodal question answering (MMQA), which aims to derive the answer from multiple knowledge modalities (e.g., text, tables, and images), received increasing attention due its board applications. Current approaches to MMQA often rely on single-modal or bi- modal QA models, which limits their abil- ity effectively integrate information across all modalities and leverage the power of pre- trained language models. To address these lim- itations, propose novel framework called UniMMQA, which unifies three different input modalities into text-to-text format by employ- ing position-enhanced table lincarization and diversified image captioning techniques. Ad- ditionally, enhance moos- reasoning by incorporating a multimodal rationale gen- erator, which produces textual descriptions of cross-modal relations for adaptation into the text-to-text generation process. Experimental results on three MMQA benchmark datasets show the superiority of UniMMQA in both su- pervised and unsupervised settings.
Corcal
Image Caplion
Lnneriiztion
Image
Text
Table
Image
Table
(a) Illustration of existing modality unification methods.
Oiparment
Date
Kitaoas
4G Plack
Coldaer
Milaed Harial
Templane based Taclo-to Toxt
Binglo Image Caption
Kazunari Murakami made his debut October 13, 1906 at Lumax Cup: Toumament of J36, losing to Alchiro Gone KO.
A view of water with town the bacliground.
(b) Examples template- based table-to-text transformation and single image caption.
Figure I: Illustration of remaining challenges in existing modality unification methods for MMQA.
ities, which failed to bridge the challenging gap between multimodal sources. To integrate in- formation across modalities, various graph struc- tures (Yang et al., 2022a; He and Wang, 2023) are introduced into MMQA systems build semantic connections between different modalities. How- ever, the graph structure is typically less compatible and practical with language models, and can barely compete with the powerful pre-trained language models (PLMs). To better leverage the knowledge from PLMs, recent studies employ modality unifi- cation techniques transform different modalities into text. For example, tabular data can be either linearized into token sequences (Xie et al., 2022: Liu et al., 2022a) or adapted to pre-defined textual templates (Yoran et al., 2022), while visual data can summarized into textual descriptions by im- age caption techniques (Gui et al., 2022; Lin et al., 2022; Yang et al., 2022b; Shao et al., 2023).
1 Introduction
typical question answering (QA) systems, an- swers are typically derived from a single modality. such as an image (Antol et al., 2015), passage (Choi et al., 2018 8), or table (Oguz et al., 2022), without the need for cross modality reasoning. However, in real-world scenarios, individuals often rely mul- timodal information and reasoning from diverse knowledge sources to arrive at answers. Multi- modal question answering (MMQA) (Hannan et al., 2020; Talmor et al., 2021) demands QA systems to perform reasoning across multiple knowledge modalities, including images, structured tables, tex- tual passages, etc.
Preliminary attempts in MMQA (Hannan et al., 2020; Talmor et al., 2021) typically utilize sepa- rate QA systems for different knowledge modal-
Despite the effectiveness of modality unification techniques handling multimodal inputs, there are several challenges that remain to be tackled
Corresponding author.

---

in MMQA: 1) Existing methods only investigate the unification of two modalities, such as text-and- table (Xie et al., 2022; Yoran et al., 2022) or text- and-image (Gui et al., 2022; Lin et al., 2022; Yang et al., 2022b; Shao et al., 2023), while an additional uni-modal QA model is required to handle the rest of questions in MMQA, as shown in Figure 1(a). 2) The process of modality transformation inevitably causes information loss. For example, the position information of each table cell is discarded when adopting template-basee based table-to-text transforma- tion, as shown in the left example of Figure 1(b). Besides, some important image details will be left out when generating single image caption, e.g., the bridge in the right example of Figure 1 1(b). 3) These MMQA approaches using modality unifica- tion typically concatenate the unified textual infor- mation as the context (Figure I(a)), while cross- modal reasoning, which attaches great importance in MMQA, is overlooked in these methods.
advantage of PLMs. Our code will be released via hittt:s:///gtthe wom/LuochhS//niiMM00
2 Related Works
Multimodal Question Answering Evolving from visual question answering (VQA) (Antol et al., 2015) that aims to answer questions from image-only inputs, knowledge-based VQA stud- ies (Marino et al., 2019; Shah et al., 2019) expand the scope to involve both textual and visual knowl- edge. Another line of studies (Wang et al., 2022) focuses on QA over a hybrid context of tabular and textual data (Chen et al., 2020b, 2021). Due to the multimodal nature of information flow in real-world applications, researchers (Hannan et al., 2020; Talmor et al., 2021; Chang et al., 2022; Li et al., 2022b) emphasize the importance of answer- ing questions that require information across multi- ple modalities, including text, tables, and images, which typically referred as multimodal question answering (MMQA). MMQA has been widely used in various real-world applications, such as finance (Zhu et al., 2021; Deng et al., 2022a), e-commerce (Deng et al., 2023, 2022h, 2020), science (Lu et al., 2022; Xu et al., 2021b), and more. Early stud- ies (Hannan et al., 2020; Talmor et al., 2021) de- compose MMQA into three single-modal QA mod- els. To align different modalities, some latest stud- ies employ graph structures (Yang et al., 2022a; He and Wang, 2023) to enhance the cross-modal interaction.
In the light of these challenges, we propose a novel MMQA framework, namely UniMMQA, which unifies three different input modalities into text-to-text format. In specific, alleviate the infor- mation loss during the modal unification, first adopt an easy-to-apply decoding strategy to gener- ate informative image captions from the visual data by sampling a diverse set of image descriptions instead of only taking the greedy one. Meanwhile, the tabular data is lincarized into textual sequence with position tokens. Then cross- modsl rationale generator is adapted produce explicit textual rea- soning descriptions about the interrelations across different modalities. All the multimodal knowledge and the cross- modal rationale are represented the text form, which are ultimately applicd to a text-to- text generation with PLMs. The main contributions of this work can be summarized as follows:
Modality Unification Pre-trained language mod- els (PLMs) show exceptional proficiency in han- dling text-to-text problems (Raffel et al., 2020), which can be further adapted into large language models (Zhao et al., 2023). Therefore, main- stream approach for handling multimodal tasks is to unify different modalities into text. As for knowledge- based VQA, a recent trend is lever- age image caption techniques for unifying the information across textual and visual modalities, which either generates captions from the whole im- age (Gui et al., 2022; Lin et al., 2022) or utilizes the information of object regions for image caption- ing (Yang et al., 2022b; Shao et al., 2023). As for table-and-text hybrid QA, some latest studies lever- age either table linearization techniques (Xie al., 2022; Liu et al., 2022a) or template-based table- to-text approaches (Yoran et al., 2022) to combine tabular and textual data. However, there has been
We propose a novel framework, namely UniM- MQA, to unify text, tables and images in MMQA into a text-to-text format for PLMs, via position- enhanced table lincarization and diversified im- age captioning.
To enhance the cross-modal reasoning over texts, we prompt a multimodal rationale generator to produce textual cross-modal relation descriptions for adapting into the text-to-text generation.
Experimental results on three MMQA bench- mark datasets show that UniMMQA outperforms existing MMQA methods and effectively takes

---

Mustimodal Pontrrt
Image:
Diversifed
aniaa
SANTA QMITA PARA
Image
kigo
Caplion
backgroand.
The santa
Text:
Input
Outpur
Rationale
Sarta Antia Park raced Santa Antaa Darby with blue logo.
Rationale:
Question Context
Text-lo-Texd PLM
Tablle:
Race
Track
Linearized table:
col: Rode Track
Santa Anita
Santa Ania
Table Lineerization
row 1: Santa Anta Derby
Derby
Park
Santa Anit Park
Jolia
Del Racetrack
row 2: La Jola Handicap
Handicap
J Del Mar Racetrack
Figure 2: The overall framework of UniMMQA.
relatively little work on unifying text, tables and images for multimodal question answering. A con- temporary work (Yu et al., 2023) also proposes to transform the images and tables into unified lan- guage representations for solving MMQA tex- tual QA problem. Nevertheless, further mitigate the information loss issues during the modality uni- fication in MMQA.
utilize an off-the-shell OCR model' to extract all explicit text in the images. Secondly, following BLIP (Li et al., 2022a), we apply a state-of-the-arl image captioning model with some modifications to transform the image into caption text from noisy web data. In order to diversify the caption genera- tion, we adopt an casy-to-apply decoding strategy by sampling diverse set of image descriptions stead of taking the greedy one (Vijayakumar et al.. 2016). The sampling strategy picks the word ac- cording to conditional probability distribution:
Image Caption Image caption (Stefanini et al., 2023) aims to generate a natural language caption summarize the objects and their attributes in- formation in a given image. In recent years, sev- eral studies utilize powerful visual-language pre- training (VLP) models to realize the captioning process (Radford et al., 2021; Li et al., 2022a; Yuan et al., 2023). Moreover, some researchers design advanced methods for generating more di- verse and informative image captions (Chen et al.. 2019; Zhao et al., 2019; Mahajan and Roth, 2020; Xu et al., 2021a). In this work, we investigate di- versified image caption techniques alleviate the information loss during modality unification.
Next
P(
1
where Itaut denotes the word in the sampling pool and denotes the next picking. In specific, we jointly employ Top-K K (Fan et al., 2018) and Top- (Holtzman et al., 2020) sampling. In Top-K sam- pling, the K most likely next words are filtered and the probability mass is redistributed for pick- ing the next word according to their cumulative probability:
En P(I
(2)
3 UniMMQA Framework
where V denotes the sampling pool. In Top-p sam- pling, the smallest possible set of words whose cumulative probability exceeds the probability p are filtered for the next word picking, which can be described as:
Given question and corresponding knowledge context = T, P), where I represents the image, T represents the table and P represents textual passage, UniMMQA aims to generate an appropriate answer A. The overview of UniMMQA is illustrated in Figure 2.
Ensree P(Itent
rtett I
2P.
(3)
3.1 Diversified Image Caption
However, Top-p sampling only filters few words in the sampling pool when the next word seems more predictable, which means it's more elegant than Top-K sampling. To handle this, we utilize
In order perform the downstream task in text space, we convert the image knowledge into text. Specifically, we employ two conversion strategies: optical character recognition (OCR) and image cap- tion. Firstly, according to Jain et al. (2021), we

---

Top-p in combination with Top-K our decod- ing strategy to avoid very low-ranked words. Our strategy then the following:
text and linearized table to the language encoder extract language representations by the following functions:
E P( 1 Mert p2pp
feature
Fi((),
(6)
K, (4)
Pfeature
Fi(Ttezt P),
(7)
where is set to x > K to prevent filtering too few words in the sampling pool. To up, feed the image into vision transformer (Dosovitskiy et al., 2021) obtain patch-level features, which are then fed into decoder to generate the image caption based on Top-K. K and Top-p sampling strategy.
where F. denotes vision extraction model CLIP and Fi denotes the language encoder implemented as a Transformer model. Then the vision features and language representations are fused to encode their joint features. The joint features are sub- sequently fed into decoder to generate rationale, which describes the interrelations across different modalities. The rationale is obtained by:
peet Feaption! (I, p),
(5)
where Feaption is a vision-language pre-trained model consists of encoder and decoder.
R = F features Pfenture);
(8)
3.2 Table-to-Tex Transformation
Besides to image, structured table also need to converted into text. A commonly used conver- sion method is linearization. In this work, we em- ploy position-enhanced encoding linearization (Liu al.,, 2022b). Specifically, we concatenate all ele- ments on the same row and different elements are separated by" "I All the rows including the header are then concatenated into a long passage delimited by predefined separator "header :" or "row: : x" where x denotes the row id. Our linearized table can be represented as:
where R denotes the rationale and F, denotes the rationale gencration model. After incorpo- rating the natural language rationale into the tex- tual input to enhance the cross-modal interaction. i.e., Ctext = {1tezt Tdezt P, P... we concate- nate it with Q as an unified input sequence X {Q1, Q2, Qn, Cfeat Cteert .... Obrart) }. where and represent the max length of question quence and context sequence respectively.
3.4 Sequence-to-Sequenee Training Procedure
Trert header: hy |..| by row 1: :: TIN
After unifying all modalities to a textual input se- quence X, we fine-tune generative PLM (e.g., T5) our QA model which is defined as:
row M: Parl ... TMNN"
Y- - Fx)(),
(9)
where M and N denote the number of rows and columns respectively. Therefore, we unify different modalities of the context C into textual sequence Crezt - [xeett Treet, P}.
where Fge is the QA model and denotes the prediction sequence. When fine-tuning on MMQA task for the answer generation stage, we minimize the negative log-likelihood loss LNL. averaged over tokens in each batch as our training objective:
3.3 Multimodal Rationale Generation
addition to unifying input modalities for text- to-text format, we generate intermediate reasoning descriptions as the rationale to seek the connec- tion and relationship between different modalities. Following Zhang al. (2023), we fine-tune gen- erative PLM (e.g., T5) on ScienceQA benchmark a rationale generator, which generates the ra- tionale based on visual features and textual inputs. Specifically, we feed the image to CLIP (Radford et al., 2021) 1) to extract vision features and we feed
1 DNLL = -
exp (Bn) (
Vi log
(10)
Σ. exp (v:)
where L is the max length of output sequence, and yi denote the I-th token in gold answer and prediction sequence respectively. Furthermore, we employ prefix-tuning as our fine-tuning strategy, which adds a specifcc prefix (e.g., MMQA) to the input sequence as a prompt.

---

Split
Modality
Test
Datasets
Train
Dev
Test
Image
Table
Text
Method
EM
EM
ManymodelQA
2,036
3,055
2,873
3,528
3,789
MultimdaaQQA
23,817
2,442
3,660
57,058
10,042
218,285
Implicit-Decome
48.8
55.5
49.3
55.9
MMCon+QA
4,582
581
590
57,058
10,042
218,285
MMQA-T5-Large
57.9
64.3
57.0
63.4
PReasM-Large
59.0
65.5
58.3
64.6
Table 1: Dataset statistics of MultimodalQQ, Many- modalQA, and MMConvQA.
SKURG
59.4
63.8
MGT
52.1
57.7
Solar
59.8
66.1
4 Experiments
UnIMMQA (5-Base)
67.9
74.0
67.0
72.9
UniMMQA (T5-Large)
71.3
77.1
75.8
4.1 Experimental Setups
UniMMQA (T5-3B)
75.5
81.7
73.7
80.1
Datasets & Evaluation Metries We conduct the experiments three public benchmark datasets, including ManymodalQA (Hannan al., 2020). MultimodalQA (Talmor et al., 2021), and MMCon- vQA (Li et al., 2022b). ManymodalQA contains 10,190 questions: 2,873 images, 3,789 text and 3,528 tables, which have been split into train/dev sets. MultimodalQA consists of 29,918 question- answer pairs cross image, table and text modali- ties, 35.7% of which require cross-modal reason- ing. Derived from MultimodalQA, MMCoivQA is multimodal conversational question answer- ing dataset with 1,179 conversations and 5,753 question-answeriing answering pairs. There are 218,285 pas- sages, 10,042 tables and 57,058 images in MMCon- vQA, and about 24.4% of conversations require reasoning cross three modalities. The statistics of these datasets presented in Table 1. Following previous studies, we adopt Exact Match (EM) and Fl as the evaluation metrics for all datasets.
Table Experimental results on MultimodalQA.
size of the answer generating process is set to 4. For training, we use batch size 4 for T5-Base and use batch size 2 for T5-Large and T5-3B due to CUDA memory. We employ the Adafactor opti- mizer for T5-Base and T5-Large, and AdamW for T5-3B with the 5e-5 initial learning rate and linear decay of the learning rate. The max training epoch 400 for all datasets. We evaluate on the de- velopment set for every 500 steps and use the aver- age development set metric for the best checkpoint selection. As for the image caption module, we jointly adopt Top-K K and Top-p sampling as decod- ing strategy in caption generating. The sampling pool size K and the probability threshold p set to K 50 and p 0.9 respectively. The number of generated sequences per image is set to N 3 for more diversified image captions.
4.2 Overall Performance
Baselines We compare UniMMQA with the state- of-the-art methods each benchmark, including Implicit-Decomp (Talmor et al., 2021). MMQA- T5 (Yoran et al., 2022), PReasM (Yoran et al., 2022), SKURG (Yang et al., 2022a), MGT (He and Wang, 2023), ManymodalQA (Hannan et al., 2020), ORConvQA (Qu et al., 2020), MAE (Li al.. 2022b), Solar (Yu al., 2023). Detailed descriptions of each baseline are provided Ap- pendix A. Our default configuration utilizes model as backbone. Specifically, we employed different sizes of T5 model to evaluate our perfor- mance, denoted as UniMMQA (T5-Base). UniM- MQA (T5-Large) and UniMMQA (T5-3B).
MultimodalQA Table 2 presents a comparison of the performance of UniMMQA on the Mul- timodalQA dataset with previous state-of-the-art models. Among the baselines, MMQA-T5 and PReasM, which unify the textual and tabular modal- ities, largely outperform Implicit-Decomy which adopts separated uni-modal and bi-modal QA models. Despite the improvement over Implicit- Decomp, SKURG and MGT, which employ graph structures to enhance the interactions between dif- ferent modalities, fail to compete with MMQA-T5 and PReasM. These results indicate the effective- ness of modality unification on the MMQA task. Furthermore, the proposed UniMMQA method sur- passes all the strong baselines on both EM and F1 by unifying all three modalities of MMQA. In ad- dition, the performance of UniMMQA also grows with the size of the backbone PLM, which shows the superiority of the proposed method to be adap-
Implementation Details We use three sizes of T5 as our backbone language models, including T5-Base, T5-Large and T5-3B. Besides, we set the max length of unified input sequence and output sequence to 1024 and 128 respectively. The beam

---

Method
EM
Dev
Test
Most Common
2.4
Method
EM
F1
EM
FI
USE
5.6
ORCon+QA
1.2
3.0
1.1
1.9
Voting
21.1
ManymodelQA
2.3
0.7
1.8
1.0
ManymodalQA
39.7
MAE
19.8
26.8
22.0
ManymodalQA (w/ Oracle)
46.3
Solar
56.8
62.5
57.3
64.6
UniMMQA (TS (T5-Base) Base)
45.4
45.7
UniMMQA (T5- Base)
57.8
64.7
59.2
64.9
UniMMQA (TS (T5-Large)
50.0
50.4
UniMMQA (T5-Large)
62.3
69.0
63.6
70.0
UniMMQA (T5-3B)
52.1
52.4
UniMMQA (T5-3B)
65.8
71.4
66.7
Table 3: Experimental results on ManymodalQA.
Table Experimental results on MMConvQA.
TS-Base
La-La
T5-3B
tive to variety of powerful PLMs.
Method
EM
F1
EM
ManymodalQAA We also conducted experiments on ManymodalQA dataset and the results are pre- sented in Table 3. Compared with heuristic meth- ods, ManymodalQA achieves much better perfor- mance by classifying the question types for dif- ferent uni-modal QA models. With oracle ques- tion type labels (w/ Oracle), ManymodalQA fur- ther improves the performance, indicating the im- portance of identifying the modalities of the re- quired information in MMQA. The results demon- strate that UniMMQA outperforms almost all base- line models the ManymodalQA dataset, but the score of UniMMQA (T5-Base) is lower than Many- modalQA (w/ Oracle).
UsiMMQA
67.9
74.0
71.3
77.1
75.5
81.7
w/o Prefix-tuning
67.6
73.7
71.0
76.8
75.3
81.4
w/o Rationale
67.9
73.9
77.0
75.4
w/o Top-k sampling
67.7
72.9
71.1
76.1
75.2
w/o Top-p sampling
67.2
76.4
75.3
w/ Beam scarch
66.1
72.6
75.6
74.5
w/ Greedy decode
66.8
72.1
71.1
75.4
75.1
80.5
w/ Template-based Table
41.6
50.9
47.7
56.5
52.1
60.0
Table 5: Ablation study MultimodalQA (Dev set).
fine-tunin fine-tuning, rationale generation, image caption- ing, and table lincarization, as presented in Table 5, There are several notable observations follows:
When dropping the prefix-tuning strategy, UniM- MQA uses common fine-tuning strategy down-stream tasks without adding a task-specific prefix into the input sequence, whose perfor- mance drops by 0.2-0.3 EM and Fl points.
MMConyQA Table summarizes the results for MMConvQA dataset, in comparison with previous SOTA. We can see that UniMMQA achieves huge improvements and outperforms the best baseline MAE by at least 39.2 EM scores and 37.0 scores. This also indicates that UniMMQA can be flexibly and effectively upplied to different types of MMQA problems, such as in the form of conversations.
When we drop the rationale in input sequence, the overall performance declines a little, which demonstrates that adding the interrelations across input modalities to input could indeed benefit en- hancing cross-modal reasoning. Although there is weak correlation between images and text MultimodalQA, our rationale generator performs well in generating relation descriptions between other highly correlated images and text. We show a corresponding example in Figure 6.
Overall UniMMQA achieves new state-of-the- art results on all MultimodalQA,, ManymodalQA and MMConvQA datasets. Compared with exist- ing methods that decompose MMQA into three single-modal models, e.g., Aaplicit-Decomy and ManymodalQA, the results show that unifying in- put modalities performs better. Besides, compare with SKURG and MGT that utilize graph struc- tures to enhance the cross modal interaction, our multimodal rationale generator is more effective and contributes to the superior results.
As for image captioning, when we drop Top-K or Top-p sampling, we can see declines in both situ- ations. When replacing with beam search (Shao al., 2017) or greedy decode (ojood((ijjyauumar al., 2016), the performance drops about 2%, which further verifies the effectiveness of jointly utiliz- ing Top-K and Top-p sampling to diversify the generated image captions.
4.3 Ablation Study
We perform ablation studies to investigate the ef- fects the proposed approaches in terms of model

---

LLaMA-7B
Vicuma-7B
LLaMA-13B
Vicuma- 13B
Prompt
EM
EM
EM
F1
EM
F1
Text + Linear. Table Image (MiniGPT-4)
2.3
13.3
1.2
14.1
Text + Template. Table + Image Caption
0.0
11.3
6.8
16.8
1.4
12.1
1.8
16.7
Text Linear. Table Image Caption
0.0
14.9
7.4
19.3
1.6
17.5
2.2
23.3
Text + Linear. Table + Diversified Image Cuption
15.0
7.6
19.3
2.4
19.0
2.4
23.7
Text + Linear. Table Diversified Image Cuption Rationale
0.0
15.3
7.8
19.4
2.4
19.2
2.6
24.1
Table 6: Zero-shot performance with LLMs MultimodalQA.
Image
Table
Text
Overall
Dataset
Model
EM
F1
HM
F1
EM
F1
FM
F1
UniMMQA (T5-Base)
66.7
69.2
65,5
71.4
68.3
76.4
67.9
74.0
MultimodalQA
UniMMQA (T5-Large)
69.8
72.2
69.9
75.6
71.9
80.0
71.3
77.1
UniMMQA (T5-3B)
73.8
76.7
73.4
79.7
76.2
84.1
75.5
81.7
UniMMQA (T5- (T5-Base)
46.6
46.9
61.1
30.2
30.4
45.4
45.7
ManymodalQA
UniMMQA (T5- (T5-Large)
48.5
48.6
67.5
68.2
34.9
35.1
50.0
50.4
UniMMQA (T5-3B)
49.8
50.2
58.0
58.3
40.9
41.3
52.1
52.4
UniMMQA (T5-Base)
73.2
75.5
33.5
40.5
66.8
76.3
57.8
64.7
MMConvQA
UniMMQA (T5-Large)
73.2
75,4
38.9
46.8
73.3
81.7
62.3
69.0
UniMMQA (T5-3B)
73.3
75.5
41.9
47.9
77.8
85.1
71.4
Table 7: Detailed performance in terms of different modalities.
For table linearization, when template- based encoding (Chen et al., 2020a; Su al.. 2021) in place of position-enhaneee encoding, we convert structured table natural language. The results show that there is a sharp decline for all model sizes, demonstrating that the template- based conversion to natural language casts neg- ative impact on the MMQA task due to the diver- sity and differences of table formats and content.
several notable observations: Multimodal LLMs (e.g.. MiniGPT-4) for encoding the orig- inal image, which demonstrates superior perfor- mance on the VQA tasks, fall short of handling the MMQA task, compared with their base LLMs with image captions. 2) Due to the instruction tuning, Vicuna performs much better than LLaMA. 3) The position-enhanced table linearization outperforms the complate-based one to a great extent. 4) Both the proposed diversified image caption technique and the generated rationale further improve the per- formance of UniMMQA. Overall, these results val- idate the applicability the proposed UniMMQA framework in the era of L.MMs.
4.4 Zero-shot Setting with LLMs
order to testify the effectiveness of the proposed framework with the use of LLMs, we evaluate the zero-shot performance of three different open- source LLMs, including LLaMA (Touvron et al., 2023), Vicuna (Chiang et al., 2023), and MiniGPT- 4 (Zhu et al., 2023). Due to the length limits in these LLMs (e.g., 2048 tokens for Vicuna), the few-shot in-context learning is impractical where the context in MMQA much longer than other tasks. Without few-shot samples, the format of the generated answers is typically a complete sentence, instead of short phrase or a text span as the refer- ence answer. Therefore, we observe that the exact match (EM) score is unreliable, and here we mainly discuss the results in terms of the Fl score.
4.5 Further Analysis
Analysis of Difffrent Modalities We analyze the performance in terms of questions requiring the in- formation from different modalities, and the results are summarized in Table As shown in the results on MultimodalQA datasets, the performance of UniMMQA on image-based and based ques- tions is observed to be lower compared to the over- all performance, whereas it performs better on text- based questions. This disparity can be attributed to the information loss that may occur during the conversion of images and tables into text, as well as the inherent challenges in effectively incorporating
Table summarizes the zero-shot performance with LLMs on the MultimodalQA dataset. There

---

visual and tabular data into the model's reasoning process. Besides, the EM and Fl on text-based and image-based questions are lower than overall performance but those on table-based questions are higher on ManymodalQA dataset. The reason is that the tables provided in ManymodalQA only con- tain few rows and its passages are too lengthy and complex. Therefore, UniMMQA is hard to make well reason from passage context but does well in table context on ManymodalQA dataset. Con- versely, the performance on table-based questions is much lower than the overall performance in MM- ConvQA, while it performs better on image-bast based and text-based questions. This indicates that it is challenging to handle conversational question an- swering over tables. Overall, we conclude that the performance the MMQA task is largely affected by the complexity of the context from each modal- ity. Therefore, worth studying approaches to effectively unifying different modalities.
EM
Overall performance
75-
60
Number of sampled image captions (N)
Figure 3: The overall performance with varying number of sampled image caption.
Question:
destination has what going acroes the
Image Caption:
Answer:
water
town
the background.
Diversifled Image Caption:
bonch avericoking fiver and
view of a city and a bridge over a fiver.
bridge.
Answer: bridoo
Applt
A bench overlooking river and bridge.
Aeall
Figure 4: Case study in terms different image caption methods.
Analysis of Number of Sampled Image Captions order to generate more diversified image caption, we set different numbers of sampled image caption N and pick the top N sequences with the highest cumulative probability. Figure 3 shows how the number of sampled image caption affects the over- all performance. We apply our best model trained on MultimodalQA and test it with varying N. As shown, the EM and F1 both increase along with the number of sampled image caption N from 65.8 to 67.9 and 72.1 to 73.9 as N increases from 1 to 3. A larger N increases the caption diversity that each image generates more than one corresponding caption. However, after N=3, the testing perfor- mance decreases along with the growing number of N. This is because setting too large N resulting in generating lengthy caption, which is hard for QA model making well reasoning.
use template- based encoding, the predefined tem- plate only seleets columns Position, Country and Athlete to generate natural language and discards column Height which is mentioned in question, thus resulting in wrong answer generation. And when using position-enhanced encoding, all the cells converted into text without any informa- tion loss, thus the model could find the correct cell and select the right answer. the example from Figure 6, when we add the multimodal rationale generator, the model could connect New York Yan- kees passage with the logo to produce their relation description and generate the right answer. When dropping the rationale, there is no connection be- tween the passage and the image, which make QA model hard to make good reasoning. Overall, we conclude that adding our modality unification mod- ule indeed effectively unifies different modalities and yields decent performance.
4.6
Case Study
To evaluate our modality unification module, we present generated examples from MultimodalQA dataset in Figure 4 to Figure 6. Figure 4 shows that when we drop out our diversified image cap- tioning strategy, the caption model fails to find the bridge in the given image and thus generates the wrong answer. When adding the strategy, the three generated captions contain more detailed in- formation the image and thus answer correctly. Figure 5 shows how different table lincarization strategy affects the answer generating. When we
Conclusion
In this paper, we present novel MMQA frame- work called UniMMQA, which unifies text, tables and images into text-to-text format for PLMs. Specifically, we first utilize an effective decoding strategy to generate diversified and informative im- age captions from the visual data. And we lin- earize the tabular data into textual sequence with

---

Question:
Which blande haired athlote competed in the IAAF Golden League of 2007 in the Women'' High Jump with height of 1.91?
performance. Therefore, it is worth studying ap- proaches to balance the informativeness and com- plexity during the modalities conversion in our fu- ture work.
Templato-base In IAAF Colden Leegue of 2007, No.
athiede la Blanka Viai from NIA. No.
Answer: Baanic Visi
athicte Palama from NA. No.
athilte Palama Marina
Altova.
Position-based:
Hallucination Issues Another limitation of UniMMQA is that our image caption and ratio- nale generation module may suffer from the typical flaw of hallucination issues, i.e., generating fab- ricated descriptions that is irrelevant to image or text. Fabricated caption and rationale would mis- lead the reasoning and result in incorrect answering. Our current approach has not taken this issue into consideration, leaving room for future research to explore and address this aspect in greater detail.
Golden League 2007
position country athlele hhight| row 1:1 NAA Blanka Viail 2.02 row 2:
Answer: marna añoov
row 3. 8 NIA] Marins Altovai 1.91
Figure 5: Case study in terms of different table lineariza- tion methods.
Question:
Among MLB tears ane cumently affiisted the Internaiinal Leagu, when did the team with a basebell bet on their logo last win World Series touraament?
The Yardowee horee wan 27 World Series Champinnsdppe
Answer (wio rationale); Philadelphia Philiies
y Lankeee
Answer rationale): New Yark Yankees
2039, under manager Jes Chord. they defeated Phi six garnes.
References
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, VQA: visual question an- swering. In 2015 IEEE International Conference on Computer Vision, ICCV 2015, pages 2425-2433.
and Devi Parikh. 2015.
Rationale:
New York Yankees have a baseball their logo.
Figure 6: Case study in terms of with and without ratio- nale generation.
Yingshan Chang, Guihong Cao, Mridu Narang, Jian- feng Gao, Hisami Suzuki, and Yonatan Bisk. 2022. Webqa: Multihop and multimodal QA. In IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, CVPR 2022. pages 16474-16483,
position tokens. Meanwhile, a cross-modal ratio- nale generator is adapted to produce explicit textual descriptions about the interrelations across differ- ent modalities. Our experiments on three MMQA benchmark datasets show that UniMMQA outper- forms many MMQA methods and effectively take advantage of PLMs.
Fuhai Chen, Rongrong Ji, Jiayi Ji, Xiaoshuai Sun, Baochang Zhang, Xuri Ge. Yongjian Wu, Feiyue Huang, and Yan Wang. 2019. Variational structured semantic inference for diverse image captioning. In Advances in Neural Information Processing Systems 32: Annual Conference Neural Information Pro- cessing Systems 2019, NeurlPS 2019, pages 1929- 1939.
Acknowledgements
Wenhu Chen, Ming-Wei Chang, Eva Schlinger. William Yang Wang, and William W. Cohen. 2021. Open question answering over tables and text. In 9th International Conference on Learning Representa- tions, ICLR 2021.
This work was supported in part by the National Natural Science Foundation of China under Grant 61602013.
Limitations
Wenhu Chen, Hongmin Wang. Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020a. Tabfact: A large-scale dataset for bable-based fact verification. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Modality Unification Although using our image captioning techniques could generate informative and diversified image captions, information gap still exists between image and caption. Specifically, our image model lacks the ability to extract very detailed information in image (e.g. the number of objects), thus resulting in wrong answering when facing some corresponding questions. Meanwhile, although using position-enhanced table lineariza- tion could convert all content into text without any information loss, it also increases the complexity of input sequence and largely affects the overall
Wenhu Chen, Hanwen Zha, Zhiyu Chen, Wenhan Xiong- Hong Wang, and William Yang Wang. 2020b. Hy- bridqa: A dataset of multi-hop question answering over tabular and textual data. In Findings of the As- sociation for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1026 1036. Association for Computational Linguistics.

---

Wei-Lin Chiang. Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
Association for Computational Linguistics: Human Language Technologies, NAACL 2022, pages 956- 968.
Darryl Hannan, Akshay Jain, and Mohit Bansal. 2020. Manymodalqa: Modality disambiguation and QA over diverse inputs. In The Thirty-Fourie AAAI Con- ference on Artificial Intelligence, AAAI 2020, pages 7879-7886.
Eunsol Choi, He He, Mohit lyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. Quac: Question answering in context. In Proceedings of the 2018 Conference on Empiri- cal Methods Natural Language Processing, Brus- sels, Belgium, October November 4, 2018. pages 2174-2184 Association for Computational Linguis- tics.
Xuehai He and Xin Wang. 2023. Multimodal graph transformer for multimodal question answering. In the European Chapter of the Association for Computational Lin- guistics, EACL 2023, pages 189-200.
Proceedings of the 17th Conference the
Yang Deng, Wengiang Lei. Wenxuan Zhang, Wai Lam, and Tat-Seng Chua. 2022a. PACIFIC; towards proae- tive conversational question answering over tabular and textual data finance. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, pages 6970- 6984.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Aman Jain, Mayank Kothyari, Vishwajeet Kumar. Preethi Jyothi, Ganesh Ramakrishnan, and Soumen Chakrabarti. 2021. Select, substitute, search: A new benchmark for Knowledge-augmentg tion answering. In SIGIR '1: The 44th International ACM SIGIR Conference on Research and Develop- ment in Information Retrieval, Virtual Event, Canada, July 11-15, 2021, pages 2491-2496 ACM.
Yang Deng, Yaliang Li, Wenxuan Zhang, Bolin Ding, and Wai Lam. 2022b. Toward personalized answer generation in e-commerce via multi- perspective pref- erence modeling. ACM Trans. Inf. Syst., 40(4):87:1- 87:28.
visual ques-
Yang Deng, Wenxuan Zhang, and Wai Lam. 2020. Opinion-award aware answer generation for review-driven question answering in e-commerce. In CIKM '20: The 29th ACM International Conference on Informa- tion and Knowledge Management, pages 255-264.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven C. H. Hoi. 2022a. BLIP: bootstrapping language-image pre-training for unified vision-language understand- ing and generation. In International Conference on Machine Learning, ICMI. 17-23 July 2022, Bal- timore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 12888-12900 PMLR.
Yang Deng, Wenxuan Zhang, Qian Yu, and Wai Lam. 2023. Product question answering in e-commerce: A survey. In Proceedings of the 61 Annual Meet- ing of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2023, pages 11951 11964.
Yongqi Li, Wenjie Li, and Liqiang Nie. 2022b. Mm- coqa: Conversational question answering over text, tables, and images. In Proceedings of the 60th An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 4220-4231. Association for Computational Linguisties.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold. Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. 2021. An image is worth 16x 16 words: Transformers for image recognition at scale. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
Yuanze Lin, Yujia Xie. Dongdong Chen, Yichong Xu, Chenguang Zhu, and Lu Yuan. 2022. REVIVE: re- gional visual representation matters in knowledge- based visual question answering. In NeurIPS.
Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages guistics.
Qian I Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, 7eqi Lin, Weizhu Chen, and Jian- Lou. 2022a. TAPEX: table pre-training via learning neural SQL executor. The Tenth International Conference on Learning Representations, ICLR 2022.
Association for Computational Lin-
Qian Liu, Bei Chen, Jiaqi Guo, Morteza Ziyadi, Zeqi Lin, Weizhu Chen, and
Liangke Gui, Borui Wang, Qiuyuan Huang, Alexan- der Hauptmann, Yonatan Bisk, and Jianfeng Gao. 2022. KAT: A knowledge augmented transformer for vision-and-lann language In Proceedings of the 2022 Conference of the North American Chapter of the
Lou. 2022b. TAPEX: table pre-training via learning a neural SQL executor. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.

---

Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai- Wei Chang, Song-Chune Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NewrlPS.
ceedings of the 2017 Conference on Empirical Meth- ods in Natural Language Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2210-2219. Association for Computational Linguis- tics.
Shweta Mahajan and Stefan Roth. 2020. Diverse im- age captioning with context-object split latent spaces. In Advances in Neural Information Processing Sys- tems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020.
Zhenwei Shao, Zhou Yu, Meng Wang. and Jun Yu. 2023. Prompting large language models with answer heuris- tics for knowledge- based visual question answering. CoRR, abs/2303.019 0.900.
Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, and Rita Cue- chiara. 2023, From show to tell: A survey on deep learning- based image captioning. IEEE Trans. Pat- tern Anal. Mach. Intell., 45(1)5539-559
Kenneth Marino, Mohammad Rastegari, Ali Farhadi. and Roozbeh Mottaghi. 2019. OK-VQA: A visual question answering benchmark requiring external knowledge. In IEÉE Conference on Computer Vi- sion and Pattern Recognition, CVPR 2019, pages 3195-3204.
Yixuan Su, David Vandyke, Sihui Wang, Yimai Fang, and Nigel Collier. 2021. Plan-then then-geneaate: Con- trolled data-to-text generation via planning. In Find- ings of the Association for Computational Linguistics: EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 16-20 November, 2021, pages 895-909. Association for Computational Linguisties.
Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Sejr Schlichtkrull. Sonal Gupta, Yashar Mehdad, and Scott Yih. 2022. Unik-qa: Unified representations of structured and unstructured knowledge for open- domain question answering. In Findings of the Asso- ciation for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pages 1546-1546. Association for Computational Linguis- tics.
Alon Talmor, Ori Yoran, Amnon Catav, Dan Lahav, Yizhong Wang, Akari Asai, Gabriel Ilharco, Han- naneh Hajishirzi, and Jonathan Berant. 2021. Mul- timodalga: complex question answering over text, tables and images. In 9th International Conference on Learning Representations, ICI. 2021.
Chen Qu, Liu Yang. Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit lyyer. 2020. Open-retrieval con- versational question answering. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 539 548. ACM.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Maric-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR. abs/2302.13971
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. 2021. Learn ing transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML. 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages $748-8763 PMLR.
Ashwin K. Vijayakumar, Michael Cogswell, Ram- prasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra. 2016. Diverse beam search: Decoding diverse solutions from neural se- quence models. CORR, abs/1610.02424.
Dingzirui Wang, Longxu Dou, and Wanxiang Che. 2022. A survey on table-and-text hybridqa: Con- cepts, methods, challenges and future directions. CoRR, abs/2212.134463 13465..
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:67 1-140:67.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang. Vic- tor Zhong, Bailin Wang. Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022. Unifiedskg: Unifying and grounding with text-to-text language models. In Pro- ceedings of the 2022 Conference on Empirical Meth- ods in Natural Language Processing. EMNLP 2022. pages 601-631.
Sanket Shah, Anand Mishra, Naganand Yadati, and Partha Pratim Taluk 2019. KVQA: knowledge- aware visual question answering. In The Thirty Third AAAI Conference on Artificial Intelligence, AAAI 2079, pages 8876-8884.
structured knowledge
Yuanlong Shao, Stephan Gouws, Denny Britz, Anna Goldie, Brian Strope, and Ray Kurzweil. 2017. Gen- erating high-quality and informative conversation re- sponses with soquence-to-seqeene models. In Pro-

---

Guanghui Xu, Shuaicheng Niu, Mingkui Tan, Yucheng Luo, Qing Du, and Qi Wu, 2021a. Towards accurate text-based image captioning with content diversity exploration. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, pages 12637- 12646.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang. Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat- Seng Chua. 2021. TAT-QA: A question answering benchmark on bybrid of tabular and textual content in finance. In Proceedings of the 59th Annual Meet- ing of the Association for Computational Linguistics and the 1ith International Joint Conference Natu- ral Language Processing, ACL/IJCNP 2021, pages 2777-3287.
Weiwen Xu, Yang Deng, Huihui Zhang. Deng Cai, and Wai Lam. 2021b. Exploiting reasoning chains for multi-hop science question answering. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 1143-1156.
A Detailed Descriptions of Baselines
Qian Yang, Qian Chen, Wen Wang, Baotian Hu, and Min Zhang. 2022a. Enhancing multi-modal and multi-hop question answering via structured knowledge and unified retrieval-generation. CoRR, abs/2212.08623
We detailedly introduce the compared baseline methods as follows:
Implicit-Decomp (Talmor et al., 2021), which consists of several single-modality and multi- modality QA models, predicts a program that specifies the required reasoning steps over dif- ferent modalities, and executes the program with different QA models.
Zhengyuan Yang, The Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. 2022b. An empirical study of GPT-3 for few-shot knowledge- based VQA. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, pages 3081 3081-3089.
Ori Yoran, Alon Talmor, and Jonathan Berant. 2022. Turning tables: Generating examples from semi- structured tables for endowing language models with reasoning skills. Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), ACL 2022, pages 6016-6031.
MMQA-T5 (Yoran et al., 2022) first fine-tunes TS (Raffel et al., 2020) on the MMQA task, with the linearized tables and text as the input. To han- dle image-based questions, Yoran et al. (2022) route those questions to Implicit-Decomee and the others to the fine-tuned T5.
Bowen Yu, Cheng Fu, Haiyang Yu, Fei Huang, and Yongbin Li. 2023. Unified language representation for question answering over text, tables, and images. In Findings of the Association for Computational Linguistics: ACL 2023, pages 4756 4765..
PReasM (Yoran et al., 2022) further fine-tunes T5 with different reasoning skills with synthetic tabular data, and then adopts the same question routing approach for handling image-hased ques- tions as MMQA-T5.
Yifei Yuan, Wenxuan Zhang, Yang Deng, and Wai Lam. 2023. Social media fashion knowledge extraction as captioning. arXiv preprint 2339.
SKURG (Yang et al., 2022a) utilizes the knowl- edge graph to integrate the multimodal inputs for modeling interdependent reasoning steps.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis. and Alex Smola. 2023. Multi- modal chain-of of thouhhttht reasoning in language mod- els. CORR, /wtp/2222.000.....
MGT (He and Wang. 2023) also employs graph structure model interactions between differ- ent modalities. MGT utilizes Transformer as the backbone and combines multimodal graph learn- ing from unstructured data with Transformers.
Sanqiang Zhao, Piyush Sharma, Tomer Levinboim, and Radu Soricut, 2019. Informative image captioning with external sources information. Proceedings of the 57th Conference of the Association for Compu- lational Linguistics, ACL 2019, pages 6485-6499 6494.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang. Xiaolei Wang. Yupeng Hou, Yingqian Min, Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, abs/2303.1822 18223.
ManymodalQA (Hannan et al., 2020) utilizes a question-type classifier to determine the modal- ity which the question belongs and directed the question and the context into the corresponding uni-modal QA models.
ORConvQA (Qu et al., 2020) is an open-retrieval conversational QA model, which consists of a learnable retriever, a reranker and a reader all based on Transformers.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023. Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, abs/2304. 10592.

---

MAE (Li al., 2022b) multimodal conversa- tional QA model (MMCoQA), which divides the MMCoQA task into three steps: conversational question understanding, multimodal evidence re- trieval, and adaptive answer extraction.
Solar (Yu et al., 2023) consists two parts: unified language representation and unified QA model. The former is responsible for transform- ing images, tables and texts in different modali- ties into language representation. The latter gen- erates answer through three steps of retrieval, ranking and decoding the language space.


Reviews:
[{"review_id": "0r1UphC7ZB", "reviewer": "Reviewer_T6GB", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and for acknowledging the quality of my paper. I sincerely appreciate the time and effort you have dedicated to reviewing my work.\n\nRegarding the performance of baselines such as MGT and SKURG on ManymodalQA and MMConvQA, we would like to clarify as follows.\n\nAs for MGT, the code has not been made available as open-source at the time of our study. Therefore, we could only rely on the performance metrics reported in their paper. Regrettably, they didn’t conduct experiments on ManymodalQA and MMConvQA datasets. As for SKURG, they first built a knowledge graph to model the relations between multi-modal sources. Specifically, they extract entities from different sources and extract their relation based on certain attributes specific to MultimodalQA. Since these attributes are not provided in the ManymodalQA and MMConvQA datasets, it is challenging to adapt this knowledge graph construction method directly to these datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper is well-written and presents a straightforward approach to improving the performance of MMQA models by aggregating multiple modals. The idea of generating intermediate reasoning descriptions to seek connections and relationships between different modalities is particularly interesting. The results are solid and suggest that this approach has potential for improving performance on MMQA datasets. It remains to be seen how baselines such as MGT and SKURG will perform on ManymodalQA and MMConvQA.", "reasons_to_reject": "NA", "questions_to_authors": "", "paper_topic_and_main_contributions": "The authors of this paper argue that current MMQA approaches are limited by relying solely on single-modal or bi-modal models. This limitation hinders their ability to effectively integrate information across all modalities and take advantage of pre-trained language models. To overcome these limitations, the authors propose a novel framework called UniMMQA. UniMMQA unifies three different input modalities into a text-to-text format using position-enhanced table linearization and diversified image captioning techniques. To further enhance cross-modal reasoning, a multimodal rationale generator is incorporated to produce textual descriptions of cross-modal relations for adaptation into the text-to-text generation process. Experimental results on three MMQA benchmark datasets demonstrate the superiority of UniMMQA in both supervised and unsupervised settings. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bJR95fuPOj", "reviewer": "Reviewer_htMw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "I would like to express my gratitude for your time and effort in reviewing our paper. We appreciate your valuable feedback and suggestions, which have prompted us to revisit our work and address the concerns raised in your evaluation. In this rebuttal, we aim to clarify certain points and provide further justification for the significance and contributions of our research.\n\n1. Regarding the potential complexity of the proposed method, we would like to clarify as follows.\n\nit is essential to note that the introduction of multiple techniques, such as image captioning and table linearization, is fundamental to effectively address the challenges posed by the multi-modal Question Answering (MMQA) task. Given the multi-modal nature of input context in MMQA task, it is necessary to extract the features of each modality. Common methods for extracting image features include image encoder and image caption, while techniques such as position-enhanced encoding and template-based encoding are used for extracting table features. These techniques are widely adopted in the existing advanced models for MMQA. And we just made some improvements to them to make them more conducive to downstream QA tasks. Therefore, compared to many existing advanced methods, our framework does not introduce excessive complexity or intricacy. \n\nWhile the introduction of cross-modal rationale generation may add some potential complexity, we firmly believe that its benefits outweigh the potential drawbacks. Our introduction of this technique, which generates textual reasoning descriptions about the interrelations across different modalities, effectively helps downstream QA tasks. Most existing work often overlooks the correlation between modalities. By introducing cross-modal rationale generation, we provide an effective means to capture and utilize such correlations. To alleviate the complexity and difficulty of implementation for practitioners, we encapsulated these techniques, allowing users to easily incorporate the desired components by providing appropriate inputs. We also want to emphasize that we will be publishing our code soon, which will further validate and support our claims.\n\n2. Regarding the comprehensive explanations and examples, we would like to clarify as follows.\n\nIn our paper, Figure 2 and Case Study show comprehensive explanations and examples to clarify the integration process. Figure 2 illustrates the convertion of image and table inputs into text through the image captioning and table linearization respectively, and the rationale generator describe the relationships between different modalities. The output of these techniques are concatenated with question to serve as the input for the downstream QA model. Figure 4-6 in Case Study provide practical examples showcasing the usage of these three techniques, including the input and output of each techniques, as well as their impact on downstream QA task. By presenting these examples, we aim to provide readers with a clear understanding of how the techniques operate together and their implications for the overall framework.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "One strength of the paper is the attempt to tackle the challenging task of multimodal question answering by unifying text, tables, and images into a text-to-text format.  Unifying text, tables, and images into a text-to-text format is an innovative approach that allows the exploitation of pre-trained language models for this task. The use of diversified image captioning and position-enhanced table linearization techniques shows an effort to effectively leverage information from multiple modalities. The incorporation of a multimodal rationale generator to produce textual descriptions of cross-modal relations is a valuable addition, as it can improve the reasoning capabilities of the UniMMQA framework.", "reasons_to_reject": "The main weakness of the paper is the potential complexity of the proposed UniMMQA framework. The introduction of multiple techniques, such as diversified image captioning, table linearization, and cross-modal rationale generation, might make the overall method intricate and difficult to implement for practitioners. The paper should include more comprehensive explanations and examples to clarify the integration process and make the framework more concise.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper addresses the task of multimodal question answering (MMQA) by proposing a novel framework called UniMMQA. The paper focuses on integrating information from text, tables, and images into a text-to-text format to leverage the power of pre-trained language models effectively. While the novelty of the paper is acceptable, the proposed method appears to be relatively complex, which may raise concerns about its practicality and ease of implementation.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bi3zZecO0p", "reviewer": "Reviewer_smpP", "summary": "", "questions": "1. As mentioned in the limitation section, \"our image model lacks the ability to extract very detailed information in image\", using image captioning model instead of raw visual features will inherently lose fine-grained information. As your main method are fine-tuning T-5 models, have you compared with plugging-in a visual encoder (or a frozen visual encoder) and fine-tuning it?\n2. Is figure-2 a good example of rationales? \"Look at the picture of the Santa Anita Park logo.\" I don't see why this rationale will help downstream QA.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "I would like to express my gratitude for your time and effort in reviewing our paper. We appreciate your valuable feedback and suggestions, which have prompted us to revisit our work and address the concerns raised in your evaluation. In this rebuttal, we aim to clarify certain points and provide further justification for the significance and contributions of our research.\n\n1. Regarding the novelty of the proposed method, we would like to clarify as follows. \n\nAs for the image-to-text convertion, we don’t just generate multiple captions but also improve the decoding strategy of the existing advanced caption model to diversify the caption generation. Specifically, we utilize Top-p in combination with Top-K as our decoding strategy and set K = 50 and p = 0.9 to reduce the multiple generated captions similarity and promote caption diversification. This strategy effectively addresses the concern of generating repetitive or similar captions. \n\nAs for the table-to-text convertion, we explored the use of template-based decoding as an alternative to our position-enhanced decoding approach. However, our experimental results, as shown in Table 5 of our paper, demonstrate that template-based decoding yielded inferior performance. In contrast, our position-enhanced decoding method not only achieves better results but also exhibits high efficiency, making it a practical and reproducible solution for table-to-text conversion. Given these compelling outcomes, we did not further investigate other advanced methods or plug-in a visual encoder, as our proposed strategies already significantly outperform the baseline. \n\nWe appreciate the reviewer's consideration of alternative approaches and further enhancements. However, based on the experimental results and the performance achieved by our proposed methods, we are confident in the efficacy of our approach. We believe that the proposed method can serve as a strong and easy-to-apply baseline for the multimodal QA problem as well as a practical solution in real-world applications, and it remains open to future research endeavors that may explore additional possibilities.\n\n2. Regarding the motivation of adopting “rationale”, we would like to clarify as follows.\n\nIn specific, we aim to generate the rationale based on visual features and textual or tabular inputs to produce textual reasoning descriptions about the interrelations across different modalities. Since the ScienceQA dataset provides intermediate reasoning across image(visual) and question(texual) as “Explanation”, our rationale generator, fine-tuned on the ScienceQA benchmark, has the capability to perform cross-modal reasoning and describe the relationships between different modalities effectively.\n\nIn figure 2, the image is the logo of Santa Anita Park, the text is about the origin of Santa Anita Park, and the table contains information about Race and Track. It is also challenging for humans to discover the correlation between three modalities, as the only common point among them is Santa Anita Park. In fact, this is related to the very low correlation between different modalities in MultimodalQA dataset. Besides, it provides only a limited number of cross-three-modal samples. Therefore, it would be difficult to present a cross-three-modal QA sample with high correlation between modalities. From a practical standpoint, the multimodal context provided by users may have high correlation. To this end, our purpose of introducing a rationale generator is to enhance reasoning across different modalities, particularly when there is a strong correlation between them. But due to the limitations of the existing datasets, we were unable to fully showcase this capability of our rationale generator in a satisfactory manner. \nHowever, there are more cases of cross-two-modal QA samples. We will add more examples for illustrating the cross-modal reasoning by the generated rationale.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Paper is structured. \n2. They conducted comprehensive experiments and ablation studies. ", "reasons_to_reject": "1. The technical novelty and contribution are very limited. They convert images into text with multiple captions, and tables into text by rows and row id. Neither provide new insights. In the literature of vision-language and table-to-text NLP, there are more advanced methods to use to convert image features and tables.\n2. They have another module which generates the \"rationale\" based on visual features and textual inputs. by fine-tuning a generative PLM on ScienceQA benchmark as a rationale generator. But again this is following previous work (Zhang et al 2023). I am also not sure to what degree one can claim this as cross-modal reasoning.", "questions_to_authors": "1. As mentioned in the limitation section, \"our image model lacks the ability to extract very detailed information in image\", using image captioning model instead of raw visual features will inherently lose fine-grained information. As your main method are fine-tuning T-5 models, have you compared with plugging-in a visual encoder (or a frozen visual encoder) and fine-tuning it?\n2. Is figure-2 a good example of rationales? \"Look at the picture of the Santa Anita Park logo.\" I don't see why this rationale will help downstream QA.", "paper_topic_and_main_contributions": "This paper studies multi-modal QA, that given input modalities (text, images, tables) the model is asked to answer questions. Their main contribution is to unify three different input modalities into text, by employing position-enhanced table linearization and diversified image captioning techniques. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "N/A", "Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work introduces a new framework, UniMMQA for multimodal QA (text, table, image), which unifies multimodal input into a text-to-text format by using table linearization and diversified image captioning techniques. On multimodalQA datasets, the proposed method outperforms prior approaches. While the method is solid and the experimental results are strong, I agree with the reviewer smpP on limited novelty---according to the author's response, the main technical novelties are combining TopP and TopK sampling at the decoding time and exploring different ways of linearizing table inputs, which sounds incremental to me as well.
Decision: Accept-Findings