SlowGuess/NAACL_2021_OCR huggingface dataset:
we collated a subset of 200 examples which have their ground truth available (collected as discussed later). We, then, apply our best performing model for these examples and generate the multimodal answers. For each of 200 pairs, we have both its predicted as well as ground truth counterparts. We conduct a human experiment where the annotators are asked to rate the quality of both textual and image part of the answer on relevance  $\mathbf{R}$  and user satisfaction  $\mathbf{S}$ . The overall quality of the answer is high if it is both relevant and provides high user satisfaction. For each pair, 5 different annotators rate the answers resulting in independent ratings for both predicted and ground truth answers. We calculate the overall quality of a predicted answer  $Q_{a}$  with respect to the ground truth by calculating the ratio between the quality (which we represent by  $\mathrm{R}^{*}\mathrm{S}$ ) of predicted answer and the ground truth answer,  $Q_{a} = \frac{R*S \text{ for predicted}}{R*S \text{ for ground truth}}$ . We compute the Pearson correlation between different metrics and  $Q_{a}$ . We observe that Rouge-1, Rouge-2, Rouge-L and BLEU yielded a correlation scores of 0.2899, 0.2716, 0.2918 and 0.2132 - indicating a moderate correlation and reassuring their viability for evaluating textual answer even in our multimodal setup. For image metrics, we found precision@1 to be most strongly correlated with human judgement (0.5421). While the expectation might be that such a metric has a perfect correlation, the user judgement is also biased by the corresponding textual answer leading to different scores even if the image is same in actual and predicted answer.
Evaluating Textual Outputs: Table 2 shows the performances of E&M against MExBERT (and its ablations) on extracting the right textual part of the multimodal answer. In order to test whether the visual attention on it's own makes any difference to the text answer quality, we also compare two variants of MExBERT - one where the visual input is zeroed out and another where the images are given as input without any supervision on the image selection. In the latter case we use the average attention weights of an image to determine its relevance to an answer. While not drastically large, we observed noticeable improvements with the visual input as compared to zero visual input, affirming our understanding about the value of utilizing multimodal input and cross-modal learning. We notice a marginal improvement in the text scores if we use proxy supervision scores during training. Intuitively, this is because of better focus of
MODEL	ROUGE-1	ROUGE-2	ROUGE-L	BLEU
E&M	46.77	43.26	47.22	25.17
MExBERT + Zero Img Input	44.10	41.90	44.91	24.28
MExBERT	45.13	43.02	45.77	24.96
MExBERT + PS	45.67	43.59	46.17	25.04
MExBERT + PS + L PT	48.12	46.22	48.82	28.01
MExBERT + PS + V PT	46.18	44.11	47.24	25.89
MExBERT + PS + V+ L PT	48.88	47.02	49.03	28.50
Table 2: Results showing the performance of E&M and MExBERT over various textual metrics for test set.
query on the target image which further enhances its attention over the correct part of the answer in the input. Due to relatively smaller corpus as compared to text only QA datasets used usually in recent works, we considered pretraining to be a natural choice to improve our model further. While the improvements in text scores with the visual training are marginal (which is expected since this training is directed at visual stream), language pretraining yields reasonable improvements as shown in Table 2.
Evaluating Image Output: We rank images in test set using our proxy supervision scores. We also select the image with the highest score as predicted by the respective model. We deem this image as Precise @1,2 or 3 depending upon if it is present in top-1, top-2 or top-3 images as ranked by our proxy-supervision mechanism. While conducting evaluation, we skip those data points which have no-image or only a single image in the input to avoid any bias in the evaluation. After removing such datapoints, there were 2,800 test datapoints with 2 or more images. As mentioned before, in the E&M, we retrieve the highest scoring image matched based on concatenation of Q, Passage P, and the extracted Answer A as the matching text, so that model has access to the whole textual input. Evidently, the results obtained are better than random but are still far from accurate. In fact, they are just more than half as good as those obtained with our heuristically created proxy scores when compared with human preferences as shown in Table 4. This shows that the problem is much harder than just using image retrieval models calling for a joint attention to understand the relevance of question, passage and answer. Using questions and answers as input text for UNITER were either poorer or similar, and hence not reported due to space limitation.
The power of joint multimodal attention is strongly evident as even without any visuo-lingual pretraining, we obtain meaningful (better than random) scores with just the averaged attention. The
The name mistral comes from the languedoc dialect of the occitan and means masterly. the same wind is called mistrau in the provençal variant of occitan. Mistra is a strong, cold, northwesterly wind that blows from southern france into the gulf of lion in the northern mediterranean, with sustained winds often exceeding  $66~\mathrm{km}/$  h (41 mph), sometimes reaching 185 km/h (115 mph). The mistral played an important part in the life and culture of Provence from the beginning. Excavations at the prehistoric site called Terra Amata at the foot of Mount Boron in Nice showed that in about 40,000 B.C. the inhab itants had built a low wall of rocks and beach stones to the northwest of their fireplace to protect their fire from the power
E & M
P:
P:
MEXBERT
The mbira is an African musical instrument consisting of a wooden board (often fitted with a resonator) with attached staggered metal tines, played by holding the instrument in the hands and plucking the tines with the thumbs. The mbira is usually classified as part of the lamellaphone family, and part of the idiophone family of musical instruments. Members of this broad family of instruments are known by a wide variety of names. In the Anglo world it is often called a thumb piano
What is Mbira
Q:
MExBERT
Figure 4: Comparison of the retrieved images for MExBERT and EM models. We observe that the joint attention mechanism incorporates a better multimodal understanding, enabling MExBERT to extract the correct images.
MODEL	PRECISION@1	PRECISION@2	PRECISION@3
Random	0.139	0.258	0.381
E&M	0.255	0.444	0.541
MExBERT	0.211	0.421	0.528
MExBERT + PS	0.268	0.449	0.544
MExBERT + PS + L PT	0.271	0.453	0.546
MExBERT + PS + V PT	0.288	0.459	0.549
MExBERT + PS + V+ L PT	0.291	0.459	0.549
Table 3: Results showing the performance of E&M and MExBERT over the image modality of the multimodal answer as measured against the proxy scores over test set
assumption, while using the highest average attention weights for selection the image, is that the model learns to focus on relevant images while being trained to optimize for better textual answer generation. Applying our proxy supervision mechanism while training the model, we find a very significant improvement specially in PRECISION @ 1 scores. PRECISION @ 2,3 scores are however similar to what we obtained with E&M. That is perhaps due to the fact that UNITER is good at establishing the relationships between text and images resulting in good PRECISION@2,3 scores but it fails at deciding the top image with high confidence due to lack of explicit understanding about where to focus on the text. Such a joint understanding is
the main strength of MExBERT. Visual pretraining yields larger improvements on PRECISION@1 metric, while the language pretraining provides marginal improvements.
Human Evaluation: While our proxy scores have been intuitively designed, they are error prone. We therefore collected human annotations over the entire test corpus to further validate our model's performance. We conduct a Mechanical Turk experiment where the turkers were asked to select an image from a given set of input images for (question, answer, source passage) triplet which embellishes the textual response. Every question-answer pair was annotated by 5 annotators, with each annotator annotating 5 such pairs; we pay $0.2 for every such annotation. We also provide an option of selecting 'no image' since some inputs might not have any relevant image that could go well with answer. We find an agreement rate of over 50% for the selected image in over 90% of the cases. We, therefore, use the average number of votes per image as a 'preference' score for the image, and use this to compute the precision values in Table 4. The performance of MExBERT against such human annotations is better than its performance when calculated over proxy scores indicate that the proposed MExBERT is robust to the noise that
MODEL	PRECISION@1	PRECISION@2	PRECISION@3
Random	0.144	0.275	0.396
E&M	0.284	0.492	0.612
MExBERT	0.196	0.385	0.498
MExBERT + PS	0.316	0.505	0.608
MExBERT + PS + L PT	0.321	0.511	0.612
MExBERT + PS + V PT	0.381	0.535	0.616
MExBERT + PS + V+ L PT	0.386	0.538	0.618
Proxy Scores	0.422	0.631	0.753
Table 4: Results comparing performance of E&M and MExBERT over the image modality of the multimodal answer based on Human Evaluation over test set
might have crept in the proxy-supervision and generalizes well. This also explains why the precision is lower in the noisy setting of proxy supervision than the low-noise setting based on the human annotations. High precision values of proxy scores over the human preference scores demonstrate the effectiveness of our proposed heuristic for preparing proxy training targets.
Machine reading comprehension and question-answering have been explored for a while, with the earliest works dating back to 1999 (Hirschman et al., 1999). Most of these works dealt with single modality at a time until recently. While earlier datasets were small, beginning with SQuAD (Rajpurkar et al., 2016) several large datasets (Rajpurkar et al., 2018; Yang et al., 2018; Choi et al., 2018; Reddy et al., 2019; Kwiatkowski et al., 2019) have been proposed. Though many of these are extractive in nature, there are a few multiple-choice datasets (Mihaylov et al., 2018; Richardson et al., 2013). Datasets like QAngaroo and HotpotQA (Welbl et al., 2018; Yang et al., 2018) enable reasoning across multiple documents. Recently, several Table-QA datasets have also been proposed, aimed at providing a natural language answer by reasoning over tables. While some datasets like WikiTableQuestions (Pasupat and Liang, 2015) and MLB (Cho et al., 2018) have natural language questions, others like TabMCQ (Jauhar et al., 2016) have multiple choice questions.
A popular exploration in multimodal question answering is Visual Question Answering or VQA (Antol et al., 2015; Goyal et al., 2017; Anderson et al., 2018; Lu et al., 2016, 2019; Tan and Bansal, 2019) where the input is a textual query along with an image and the output is a text answer. Another variant of this, Charts Question Answering (Kafle et al., 2020, 2018; Kahou et al., 2017; Chaudhry
et al., 2020), allows for the input to be a chart instead of a natural image. While both of these problems involve multimodality (image + question or chart + question), the output is still textual (specifically an answer class since this is modelled as a classification problem usually). While the question is received as a text in these problems, the reasoning is performed over a single modality only. In our work, we reason out across multimodal input by simultaneously attending to images and text in the input to arrive at our target output.
To overcome unimodal reasoning, there are attempts at truly multimodal reasoning with the datasets such as ManyModalQA (Hannan et al., 2020), RecipeQA(Yagcioglu et al., 2018), and TVQA (Lei et al., 2018). While RecipeQA aims reasoning over recipes and the associated pictures, TVQA involves multimodal comprehension over videos and their subtitles. The recently proposed ManyModalQA goes a step further by adding tables to the multimodal reasoning as well. However, these datasets provide responses in a single modality only, either an MCQ or textual response. With the rate at which multimodal consumption is taking place in our lives, it is important that the answering systems also enable multimodal output which, as discussed, already can provide better cognitive understanding when combined with textual modality.
We presented one of the first exploration, to the best of our knowledge, of multimodal output question answering from multimodal inputs and proposed usage of publicly available textual datasets for it. We proposed strong baselines by utilizing the existing frameworks for extract textual answers and independently match them with an appropriate image. We demonstrate the value of a joint-multimodal understanding for multimodal outputs in our problem setup by developing a multimodal framework MExBERT which outperformed the baselines significantly on several metrics. We also developed a proxy supervision technique in absence of labelled outputs and showed its effectiveness for improved multimodal question answering. We used some existing metrics to compare the different models and justified the usage of these metrics based on a human experiment.
While it is an interesting and challenging task even in its current shape, we believe there are several limitations in our proposed framework. While
our datasets had multimodal elements, modeling multimodal reasoning from multimodal inputs and using it to arrive at a multimodal answer calls for a more careful question curation that includes these challenges. Recently proposed datasets such as MultimodalQA have created questions explicitly aimed at reasoning across multimodal input, but however, lack the multimodal output component. Future works could include questions which specifically aim for a visual elements making the output requirement multimodal. Also, free form answer generation in the multimodal input/output context is another interesting subject of further research.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6077-6086.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. 2015. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pages 2425-2433.
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, et al. 2020. Experience grounds language. arXiv preprint arXiv:2004.10151.
Ritwick Chaudhry, Sumit Shekhar, Utkarsh Gupta, Pranav Maneriker, Prann Bansal, and Ajay Joshi. 2020. Leaf-qa: Locate, encode & attend for figure question answering. In The IEEE Winter Conference on Applications of Computer Vision, pages 3512-3521.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019. Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909.11740.
Minseok Cho, Reinald Kim Amplayo, Seung-won Hwang, and Jonghyuck Park. 2018. Adversarial tableqa: Attention supervision for question answering on tables. arXiv preprint arXiv:1810.08113.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettle-moyer. 2018. Quac: Question answering in context. arXiv preprint arXiv:1808.07036.
Edgar Dale. 1969. Audiovisual methods in teaching.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. 2018. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 2054-2063.
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 326-335.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Haytham M Fayek and Justin Johnson. 2020. Temporal reasoning via audio question answering. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6904-6913.
Darryl Hannan, Akshay Jain, and Mohit Bansal. 2020. *Manymodalqa: Modality disambiguation and qa over diverse inputs*. In *AAAI*, pages 7879-7886.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in neural information processing systems, pages 1693-1701.
Lynette Hirschman, Marc Light, Eric Breck, and John D Burger. 1999. Deep read: A reading comprehension system. In Proceedings of the 37th annual meeting of the Association for Computational Linguistics, pages 325-332.
Sujay Kumar Jauhar, Peter Turney, and Eduard Hovy. 2016. Tables as semi-structured knowledge for question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 474-483.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. 2018. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5648-5656.
Kushal Kafle, Robik Shrestha, Scott Cohen, Brian Price, and Christopher Kanan. 2020. Answering questions about data visualizations using efficient bimodal fusion. In *The IEEE Winter Conference on Applications of Computer Vision*, pages 1498-1507.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua
Bengio. 2017. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300.
Tomáš Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466.
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. 2018. Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visi-olinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems, pages 13-23.
Jiasen Lu, Jianwei Yang, Dhruv Batra, and Devi Parikh. 2016. Hierarchical question-image co-attention for visual question answering. In Advances in neural information processing systems, pages 289-297.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Xilin Chen, and Ming Zhou. 2020. Univilm: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789.
Roxana Moreno and Richard Mayer. 2007. Interactive multimodal learning environments. Educational psychology review, 19(3):309-326.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human-generated machine reading comprehension dataset.
Panupong Pasupat and Percy Liang. 2015. Compositional semantic parsing on semi-structured tables. arXiv preprint arXiv:1508.00305.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. arXiv preprint arXiv:1606.05250.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. Transactions of the Association for Computational Linguistics, 7:249-266.
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mtest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193-203.
Ramon Sanabria, Ozan Caglayan, Shruti Palaskar, Desmond Elliott, Loic Barrault, Lucia Specia, and Florian Metze. 2018. How2: a large-scale dataset for multimodal language understanding. arXiv preprint arXiv:1811.00347.
Michael Sankey, Dawn Birch, and Michael Gardiner. 2010. Engaging students through multimodal learning environments: The journey continues. In Proceedings ASCILITE 2010: 27th annual conference of the Australasian Society for Computers in Learning in Tertiary Education: curriculum, technology and transformation for an unknown future, pages 852-863. University of Queensland.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565.
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556.
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv:1908.07490.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302.
Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. 2018. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes. arXiv preprint arXiv:1809.00812.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600.
Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang, and Chengqing Zong. 2018. Msmo: Multimodal summarization with multimodal output. In Proceedings of the 2018 conference on empirical methods in natural language processing, pages 4154-4164.
Our models were trained on 4 V100 machines and takes just 1 sec for the whole people in such setting. As mentioned, Except pretraining experiments and baseline experiments, all our experiments on MExBERT have been conducted with 3 random seeds and the re-ported scores have been averaged over the 3 seeds. For pretraining, we use an Adam optimizer with a learning rate of 0.0001 for 2 Epochs over 3.2 mil-lion Image-Text pairs for all our ablations during pretraining stage. We use768dimensional textual embeddings with a vocabulary size of 30,522and intermediate hidden embedding size 3072for both textual and visual features. We project 4096-dimensional VGG-19 image features into 2048 dimensions and use it as input to the visual stream