SlowGuess/NAACL_2021_OCR huggingface dataset:
Hrituraj Singh
Triomics,
Anshul Nasery, Denil Mehta, Aishwarya Agarwal, Jatin Lamba
Bombay, India
Balaji Vasan Srinivasan
Adobe Research,
Multimodal research has picked up significantly in the space of question answering with the task being extended to visual question answering, charts question answering as well as multimodal input question answering. However, all these explorations produce a unimodal textual output as the answer. In this paper, we propose a novel task - MIMOQA - Multimodal Input Multimodal Output Question Answering in which the output is also multimodal. Through human experiments, we empirically show that such multimodal outputs provide better cognitive understanding of the answers. We also propose a novel multimodal question-answering framework, MExBERT, that incorporates a joint textual and visual attention towards producing such a multimodal output. Our method relies on a novel multimodal dataset curated for this problem from publicly available unimodal datasets. We show the superior performance of MExBERT against strong baselines on both the automatic as well as human metrics.
Multimodal content is at the heart of digital revolution happening around the world. While the term modality has multiple connotations, one of its common usage is to indicate the content modality i.e. images, text, audio etc. It has been shown that multimodal content is more engaging and provides better cognitive understanding to the end user (Dale, 1969; Moreno and Mayer, 2007; Sankey et al., 2010). With recent improvements in vision-language grounding and multimodal understanding (Bisk et al., 2020; Luo et al., 2020; Sanabria et al., 2018; Das et al., 2018), several works have explored beyond unimodal machine comprehension (Hermann et al., 2015; Kočisky et al., 2018; Nguyen et al., 2016; Kwiatkowski et al., 2019) towards a holistic multimodal comprehension (Antol
et al., 2015; Das et al., 2017; Anderson et al., 2018; Zhu et al., 2018; Goyal et al., 2017; Fayek and Johnson, 2020) with significant improvements.
However, all these explorations on multimodal understanding, question answering in particular, have limited their focus to unimodal outputs even with multimodal inputs. For example - Visual Question Answering (VQA) task takes a textual query and an image to produce a textual answer. The multimodal question answering tasks (Antol et al., 2015; Kafle et al., 2018; Lei et al., 2018) take multiple input modalities, but the output is limited to text only. Even the recently proposed ManyModalQA (Hannan et al., 2020) relies on multimodal understanding to produce a textual answer. These works implicitly assume that the textual answers can satisfy the needs of the query across multiple input modalities. We posit that such an assumption is not always true; while textual answers can address several queries, a multimodal answer almost always enhances the cognitive understanding of the end user; understanding the answer through visuals is faster and provides enhanced user satisfaction.
In this paper, we propose a new task, Multimodal Input Multimodal Output Question Answering (MIMOQA), which not only takes multimodal input but also answers the question with a multimodal output. Our key contributions are:
1) We introduce the problem of multimodal input multimodal output question answering. We establish the importance of such multimodal outputs in question-answering for enhanced cognitive understanding via human experiments.
2) We propose MExBERT, a novel multimodal framework for extracting multimodal answers to a given question and compare it against relevant strong baselines. Our proposed method includes a novel pretraining methodology and uses a proxy supervision technique for the image selection.
3) We curate a large dataset for the introduced prob
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5317-5332 June 6-11, 2021. ©2021 Association for Computational Linguistics
lem by extending the MS-MARCO (Nguyen et al., 2016) and Natural Question (Kwiatkowski et al., 2019) datasets to account for multimodal outputs. We propose the use of different automatic metrics and conduct human experiments to show their effectiveness.
2 Multimodal Output
Multimodal output not only provides better understanding to the end user but also provides grounding to the actual answer. For e.g., the multimodal output for the question in Figure 1(a) aids in better comprehension of the answer, while also providing grounding to words like 'stick', 'knob'. In some cases, textual answer might even be insufficient, especially, for questions which seek explicit visual understanding (questions about colors, structures, etc). In such cases, existing systems apply image understanding on top of the images to arrive at a 'textual description' of the desired answer. While this might suffice in some cases, a multimodal output can almost always enhance the quality of such answers. In Fig. 1(b), the textual answer is insufficient and gets completed only with the help of the final image-text combination.
What is a Shillelagh?
A wooden walking stick and club or cudgel typically made from a stout knotty stick with a large knob at the top.
(A)
Figure 1: (a): The textual answer is sufficient but images provide better understanding, (b) The textual answer is insufficient and is completed by an image
How does Coronavirus look?
They have characteristic club-shaped spikes that project from their surface,
(B)
To verify the hypothesis, we collated 200 Question-Answer pairs (refer to supplementary for details); for each pair, we created its unimodal and multimodal answers. We conducted a human experiment where each question-answer pair was judged by 5 annotators; each annotator rating if the textual answer is sufficient for the input query. Irrespective of its sufficiency, the annotators were also asked whether the image in the multimodal variant enhances the understanding of the answer and adds value to it. To avoid the natural bias towards richer multimodal response in such experiments, we had explicitly inserted a few questions with irrelevant images  $(20\%)$  and only considered the annotations which did not exhibit any bias in such questions.
Out of  $80.27\%$  of the total responses where the
annotators felt that textual answers were sufficient,  $87.5\%$  felt the image enhanced their understanding even with such sufficient textual answer validating the importance of a multimodal answer. However, only  $22.2\%$  of the annotators felt the same when an irrelevant image was shown, indicating the absence of a strong bias towards richer responses. When the text was insufficient (19.73% of the responses), the relevant image boosted the understanding in  $90.62\%$  of the cases, further indicating that text only answers are not always sufficient and in such cases, an appropriate image can aid in better understanding. Here again, only  $27.65\%$  felt that an irrelevant image will add such a value, again indicating the lack of a strong bias towards multimodal answers just because they are richer. This experiment establishes that multimodal answers almost always improves the overall understanding irrespective of the sufficiency of textual answer. Motivated by this, we propose the novel problem of multimodal input, multimodal output (MIMO) QA - which attends to multiple modalities and provides responses in multiple modalities.
3 Multimodal Output QA
Formally, given a piece of input text  $\mathbf{T}$  along with a set of related images  $\mathbf{I}$  and a query  $\mathbf{Q}$ , our problem is to extract a multimodal answer  $\mathbf{M}$  from  $\{\mathbf{I}, \mathbf{T}\}$ . In an ideal case, multimodal answers do not have to be multi-modal, especially when there is no relevant image in the input. However, for the sake of simplicity, we assume that there is at least one image in the input that can complement the textual answer even if the image is not extremely critical to the textual answer for it to make sense. This follows our human experiments which showed that image adds value to the response over 90% of the time, irrespective of the sufficiency of the textual answers. Thus, our multimodal answer  $\mathbf{M}$  consists of a text  $\mathbf{M}_{\mathrm{T}}$  and an accompanying image  $\mathbf{M}_{\mathrm{I}}$ .
Multimodal Extractive BERT (MExBERT): As we show later, a major problem with independently extracting the textual answer and matching an image is the absence of joint understanding of visual and textual requirements for the query. We, therefore, propose a joint attention Multimodal Extractive BERT based framework (MExBERT) using query  $\mathbf{Q}$  over both input text  $\mathbf{T}$  and input images I. Figure 2 shows the overall architecture of our proposed MExBERT framework. Inspired by the recent visuo-lingual models (Tan and Bansal,
Figure 2: MExBERT. Details of the three blocks in the visual and textual streams are illustrated on the top. The visual stream takes the output of VGG-19 as input while the textual stream takes BERT Embeddings as input
2019; Lu et al., 2019; Chen et al., 2019), our framework has two separate streams - textual and visual stream; textual stream takes the query and input passage as input while visual stream takes the images as input.
The textual stream is extended from the BERT-QA framework (Devlin et al., 2018) and consists of self-attention transformer (Vaswani et al., 2017) layers. The input to the textual stream as shown in Figure 2 is tokenized BERT embedding of words in both passage and query. We also use the standard [CLS] and [SEP] tokens - the former prepended in the beginning and the latter embedded between query and the input passage. We use positional embedding to additionally provide positional and segment information for the MExBERT to better distinguish between query and passage. Unlike the canonical BERT-QA, our textual stream employs two types of layers - regular self-attention layers and additional cross-attention layers. The initial layers of the textual stream include  $N_{T_a}$  regular self-attention based transformer layers similar to the canonical BERT-QA. The latter half of the textual stream is composed of  $N_{T_b}$  layers each of which consists of an additional cross-attention block along with the regular self-attention. Representing the attention computation in query-key-value format, the cross-attention block uses textual tokens as query and image representation from the visual stream as keys and values. This is different
from self-attention where (query, keys and values) are all input textual tokens of the textual stream. The cross-attention block enables the framework to choose spans that are also coherent with the visual stream. If the  $i^{th}$  textual token's features and  $j^{th}$  image's features used as input for  $k^{th}$  textual stream layer and  $(k - N_{T_a})^{th}$  visual stream layer (as discussed later) are given by  $T_{k - 1}^i$  and  $V_{k - 1}^j$ ; attention with  $q$  query,  $k$  keys, and  $v$  values is  $attn(q,k,v)$ , the self-attention and cross-attention is given by,
$$
T _ {k _ {s e l f}} ^ {i} = \operatorname {a t t n} \left(T _ {k - 1} ^ {i}, T _ {k - 1}, T _ {k - 1}\right), \tag {1}
$$
$$
T _ {k _ {c r o s s}} ^ {i} = \operatorname {a t t n} \left(T _ {k _ {s e l f}} ^ {i}, V _ {k - 1}, V _ {k - 1}\right) \tag {2}
$$
where  $T_{k}:\{T_{k}^{0},\dots,T_{k}^{n}\}$  and  $V_{k}:\{V_{k}^{0},\dots,V_{k}^{m}\}$ . Here,  $n$  is the number of textual tokens and  $m$  is the number of input images. The final layer of the textual stream is used to calculate the start and end position of the answer, similar to the canonical BERT-QA (Devlin et al., 2018) where one linear layer predicts the starting token and another layer predicts ending token through softmax applied over all tokens. The goal is to optimize the cross entropy loss over both the token position predictions.
The visual stream is similar to the textual stream with two key differences - (i) There is only one type of layer in the network and the number of layers  $N_V = N_{T_b}$  and (ii) All the layers consist of only cross-attention blocks (along with feedforward layers and residual connections) and do not contain self-attention block as shown in Figure 2. The self-attention was not used as the images mostly derive their relevance/context from the textual counterparts (powered by the cross-attention) in the input passage or query rather than other input images. The cross-attention is similar to the textual stream except that query is an image feature vector and the keys and values are textual tokens' representation from the corresponding textual stream layer. The input to the visual stream is the global VGG-19 (Simonyan and Zisserman, 2014) features of each of the images. We do not use positional/segment encodings in the visual stream. We use a linear head on top of visual features to predict whether a particular image should be in the output answer and use weighted binary cross-entropy for training where the weights  $w$  and  $1 - w$  come from the proxy supervision values (as discussed later). The image with the highest confidence score on inclusion in the answer is regarded as the predicted image during inference.
Extract & Match: A natural framework to output a multimodal response would be to combine existing state-of-the-art frameworks in question answering and visuo-lingual understanding. To illustrate the shortcomings of such an assembled framework and motivate the need for a holistic framework, we implement such a framework using existing models as our Extract & Match baseline. Given the input query (Q) and the input text (T) and images (I), we first extract the textual answer using unimodal BERT-QA. (Devlin et al., 2018). We use this extracted answer, query, and input text to select an image from the input images using UNITER (Chen et al., 2019) to rank the images. UNITER has been trained on millions of image-text pairs for image-text matching task - the task of identifying whether a given image-text pair are actually the image and its caption. Due to strong pretraining, UNITER has achieved SOTA performance on a variety of vision and language task, including zero shot image-text matching. So, we use this as our baseline for image selection. We provide each image along with the text (answer, query and input) to UNITER and use the classification confidence predicted by image-text matching head to rank the images. The image which receives the highest confidence score for a given text is taken as the matched output.
4 Dataset & Pretraining
Since there is no existing dataset which satisfies the requirements of the task, we curate a new dataset (refer to supplementary for details on curation strategy and data samples) by utilizing the existing public datasets. We observe that several QA datasets contain answers that come from a Wikipedia article. Since most Wikipedia articles come with a set of related images, such images could feature as the input I in our setup. Extending this heuristic, we use two QA datasets - MS-MARCO (Nguyen et al., 2016) and Natural Question (NQ) (Kwiatkowski et al., 2019), to extract those question-answer pairs which are originally extracted from Wikipedia and scrape all images from the original article. More details about the curation process and examples of the images scraped for questions can be found in the appendix.
Table 1 shows various statistics about the dataset. The dataset includes large number of images making the task of selecting appropriate image nontrivial. The variety of images also necessitates a robust visual and language understanding by our
model. The passages have been formed by combining the answer source passage and randomly chosen  $2 - 3$  'distractor' passages from the original Wikipedia article. This allows the model to learn to find the right answer in unseen conditions also. The # of tokens in our input passages are large enough to be regarded to as a full input (instead of using the entire article) considering the focus here is on multimodal output and not article-passage ranking.
Proxy Supervision: Although we have scraped the images from the original articles, we do not have any supervision for these images in our dataset. In order to train the model to judge which images are relevant to an answer, we heuristically compute proxy targets by using two types of information about the image - its position in the original article and its caption. We use the caption and position information only to obtain the target scores during training and not as an explicit input to our model since such information is not always readily available. Thus, our model is able to infer the correct multimodal response irrespective of the availability of such information at inference time. Since MS-MARCO and Natural Questions provide information about the original source passage for the final answer, we know the position of the source passage. We calculate the proximity distance  $\mathbf{P}$  between the first token of source passage of answer and an image with number of tokens chosen as the distance unit. We, further, normalize this with the total number of tokens present in the entire article. We calculate the TF-IDF similarity of the caption against the Query, Answer and source passage (Figure 3). The overall supervision score is calculated as a weighted sum of these 4 scores where proximity score is calculated as  $1 - P$ . The normalized supervision scores (between  $0 - 1$ ) are used as targets for linear layer of the visual stream.
Pretraining: Vision and Language Tasks have relied on pretraining to address the complexities in building visuo-lingual relationships (Tan and Bansal, 2019; Lu et al., 2019; Chen et al., 2019). Following this, we leverage pretraining to better initialize our model. Further, our signals (even after including proxy supervision) are relatively
# of pairs	Avg # of tokens	# of Images
Train	52,466	242.31	373,230
Development	722	180.62	3,563
Test	3,505	242.58	24,389
Table 1: Statistics for the all three different splits of the curated MIMO Question Answering Dataset
Figure 3: Calculation of proxy supervision scores for an image. We compute the TF-IDF similarities of the caption with the question, answer and relevant paragraph, and also compute the distance of the image from the answer paragraph. These are summed in a weighted fashion to get the final score.
sparse for a visuo-lingual task, calling for a stronger model initialization. We use Conceptual Captions (Sharma et al., 2018) as it has been shown to impart a generic V-L understanding (Chen et al., 2019). We use the standard Masked Language Modelling (MLM) task over the Conceptual Captions to pretrain the textual stream and employ the cross entropy loss over the masked tokens. While the task is intended to train the textual stream, since the entire caption is generated from the visual information through the cross-attention mechanism, visual stream is also fine-tuned in this process. Since, our final model uses segment IDs, we randomly assign a segment ID of either query or passage to each caption during pretraining in order to imbibe language understanding for both type of tokens. For pretraining the visual stream, we modify the Conceptual Captions (Sharma et al., 2018) by choosing a random number between  $(3 - 10)$  (N) for each caption followed by selecting N-1 negative images (i.e. those images which have different captions) along with the image that is associated with the caption. We provide the caption as input to the textual stream and these N images as input to the visual stream. We train the model to predict the image corresponding to the caption by using binary cross entropy loss over images. Again, while this tasks is focused majorly on visual stream initialization, the textual stream is also fine-tuned due to the cross-attention layers between the two streams.
We conduct extensive experiments and ablations for the proposed MExBERT framework and compare it against the E&M baseline. We divide our curated dataset into train, development and test sets as shown in Table 1. As mentioned before, we used the 3.2 million Image-Caption pairs from
Conceptual Captions dataset (Sharma et al., 2018) for pretraining MExBERT layers. For proxy supervision, we empirically determine the weights: the proximity weight  $w_{px} = 0.4$ , passage weight  $w_{p} = 0.3$ , query weights  $w_{q} = 0.15$  and answer weight  $w_{a} = 0.15$  after analyzing the manually selected images in the dev set (as discussed later).
For the E&M baseline, we pretrain the text extraction with the SQUAD dataset (Rajpurkar et al., 2016) and finetune it on our dataset. For the image matching, we use image ranking using the input query (Q), input passage  $\mathbf{P}$  and the extracted input answer A all concatenated together. For MExBERT, we tested different variants with and without proxy supervision (PS); with different pre-training setups - pretraining the textual stream alone, visual stream alone and both - to test the independent value of different pre-training.
Except pretraining experiments and baseline experiments, all our experiments on MExBERT have been conducted with 3 random seeds and the reported scores have been averaged over the 3 seeds. We use BERT pretrained embeddings for the textual stream of MExBERT and use  $N_{T_a} = N_{T_b} = N_V = 6$ . For finetuning MExBERT, we use Adam optimizer initialized with a learning rate of 0.0001 and train it till the validation loss saturates. The model was trained over 4 V100 machines using a batch size of 8 for finetuning and 64 for pretraining. For pretraining, we use an Adam optimizer with a learning rate of 0.0001 for 2 Epochs over 3.2 million Image-Text pairs for all our ablations during pretraining stage. We use 768 dimensional textual embeddings with a vocabulary size of 30, 522 and intermediate hidden embedding size 3072 for both textual and visual features. We project 4096 dimensional VGG-19 image features into 2048 dimensions and use it as input to the visual stream.
Evaluation Metrics: We independently evaluate the text and image part of the extracted answer using various metrics. For the text, we considered standard metrics like ROUGE, BLEU popularly used in the literature for textual question answering task. For images, we use the precision @1,2 and 3 in which we measure if the predicted image is in top-1,2 or 3 images as selected in the ground truth. Although these metrics are standard, we verify their utility in the multi-modal case by conducting a human experiment and calculating their correlations with human judgments.
To further validate the choice of our metrics,