SlowGuess/ACL_2020_OCR huggingface dataset:
Chen Zheng, Quan Guo, Parisa Kordjamshidi
Department of Computer Science and Engineering, Michigan State University
East Lansing, MI 48824
{zhengc12, guoquan, kordjams}@msu.edu
This work deals with the challenge of learning and reasoning over language and vision data for the related downstream tasks such as visual question answering (VQA) and natural language for visual reasoning (NLVR). We design a novel cross-modality relevance module that is used in an end-to-end framework to learn the relevance representation between components of various input modalities under the supervision of a target task, which is more generalizable to unobserved data compared to merely reshaping the original representation space. In addition to modeling the relevance between the textual entities and visual entities, we model the higher-order relevance between entity relations in the text and object relations in the image. Our proposed approach shows competitive performance on two different language and vision tasks using public benchmarks and improves the state-of-the-art published results. The learned alignments of input spaces and their relevance representations by NLVR task boost the training efficiency of VQA task.
Real-world problems often involve data from multiple modalities and resources. Solving a problem at hand usually requires the ability to reason about the components across all the involved modalities. Examples of such tasks are visual question answering (VQA) (Antol et al., 2015; Goyal et al., 2017) and natural language visual reasoning (NLVR) (Suhr et al., 2017, 2018). One key to intelligence here is to identify the relations between the modalities, combine and reason over them for decision making. Deep learning is a prominent technique to learn representations of the data for decision making for various target tasks. It has achieved supreme performance based on large scale corpora (Devlin et al., 2019). However, it is a
challenge to learn joint representations for cross-modality data because deep learning is data-hungry. There are many recent efforts to build such multimodality datasets (Lin et al., 2014; Krishna et al., 2017; Johnson et al., 2017; Antol et al., 2015; Suhr et al., 2017; Goyal et al., 2017; Suhr et al., 2018). Researchers develop models by joining features, aligning representation spaces, and using Transformers (Li et al., 2019b; Tan and Bansal, 2019). However, generalizability is still an issue when operating on unobserved data. It is hard for deep learning models to capture high-order patterns of reasoning, which is essential for their generalizability.
There are several challenging research directions for addressing learning representations for cross-modality data and enabling reasoning for target tasks. First is the alignment of the representation spaces for multiple modalities; second is designing architectures with the ability to capture high-order relations for generalizability of reasoning; third is using pre-trained modules to make the most use of minimal data.
An orthogonal direction to the above-mentioned aspects of learning is finding relevance between the components and the structure of various modalities when working with multi-modal data. Most of the previous language and visual reasoning models try to capture the relevance by learning representations based on an attention mechanism. Finding relevance, known as matching, is a fundamental task in information retrieval (IR) (Mitra et al., 2017). Benefiting from matching, Transformer models gain the excellent ability to index, retrieve, and combine features of underlying instances by a matching score (Vaswani et al., 2017), which leads to the state-of-the-art performance in various tasks (Devlin et al., 2019). However, the matching in the attention mechanism is used to learn a set of weights to highlight the importance of various components.
7642
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7642-7651 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
In our proposed model, we learn representations directly based on the relevance score inspired by the ideas from IR models. In contrast to the attention mechanism and Transformer models, we claim that the relevance patterns are as important. With proper alignment of the representation spaces of different input modalities, matching can be applied to those spaces. The idea of learning relevance patterns is similar to Siamese networks (Koch et al., 2015) which learn transferable patterns of similarity of two image representations for one-shot image recognition. Similarity metric between two modalities is shown to be helpful for aligning multiple spaces of modalities (Frome et al., 2013).
The contributions of this work are as follows: 1) We propose a cross-modality relevance (CMR) framework that considers entity relevance and high-order relational relevance between the two modalities with an alignment of representation spaces. The model can be trained end-to-end with customizable target tasks. 2) We evaluate the methods and analyze the results on both VQA and NLVR tasks using VQA v2.0 and  $\mathrm{NLVR}^2$  datasets respectively. We improve state-of-the-art on both tasks' published results. Our analysis shows the significance of the patterns of relevance for the reasoning, and the CMR model trained on  $\mathrm{NLVR}^2$  boosts the training efficiency of the VQA task.
Language and Vision Tasks. Learning and decision making based on natural language and visual information has attracted the attention of many researchers due to exposing many interesting research challenges to the AI community. Among many other efforts (Lin et al., 2014; Krishna et al., 2017; Johnson et al., 2017), Antol et al. proposed the VQA challenge that contains open-ended questions about images that require an understanding of and reasoning about language and visual components. Suhr et al. proposed the NLVR task that asks models to determine whether a sentence is true based on the image.
Attention Based Representation. Transformers are stacked self-attention models for general purpose sequence representation (Vaswani et al., 2017). They have been shown to achieve extraordinary success in natural language processing not only for better results but also for efficiency due to their parallel computations. Self-attention is a mechanism
to reshape representations of components based on relevance scores. They have been shown to be effective in generating contextualized representations for text entities. More importantly, there are several efforts to pre-train huge Transformers based on large scale corpora (Devlin et al., 2019; Yang et al., 2019; Radford et al., 2019) on multiple popular tasks to enable exploiting them and performing other tasks with small corpora. Researchers also extended Transformers with both textual and visual modalities (Li et al., 2019b; Sun et al., 2019; Tan and Bansal, 2019; Su et al., 2020; Tsai et al., 2019). Sophisticated pre-training strategies were introduced to boost the performance (Tan and Bansal, 2019). However, as mentioned above, modeling relations between components is still a challenge for the approaches that try reshaping the entity representation space while the relevance score can be more expressive for these relations. In our CMR framework, we model high-order relations in relevance representation space rather than the entity representation space.
Matching Models. Matching is a fundamental task in information retrieval (IR). There are IR models that focus on comparing the global representation matching (Huang et al., 2013; Shen et al., 2014), the local components (a.k.a terms) matching (Guo et al., 2016; Pang et al., 2016), and hybrid methods (Mitra et al., 2017). Our relevance framework is partially inspired by the local components matching which we apply here to model the relevance of the components of the model's inputs. However, our work differs in several significant ways. First, we work under the cross-modality setting. Second, we extend the relevance to a high-order, i.e. model the relevance of entity relations. Third, our framework can work with different target tasks, and we show that the parameters trained on one task can boost the training of another.
3 Cross-Modality Relevance
Cross-Modality Relevance (CMR) aims to establish a framework for general purpose relevance in various tasks. As an end-to-end model, it encodes the relevance between the components of input modalities under task-specific supervision. We further add a high-order relevance between relations that occur in each modality.
Figure 1 shows the proposed architecture. We first encode data from different modalities with single modality Transformers and align the encoding
7643
Figure 1: Cross-Modality Relevance model is composed of single-modality transformer, cross-modality transformer, entity relevance, and high-order relational relevance, followed by a task-specific classifier.
spaces by a cross-modality Transformer. We consistently refer to the words in text and objects in images (i.e. bounding boxes in images) as "entities" and their representations as "Entity Representations". We use the relevance between the components of the two modalities to model the relation between them. The relevance includes the relevance between their entities, as shown in the "Entity Relevance", and high-order relevance between their relations, as shown in the "Relational Relevance". We learn the representations of the affinity matrix of relevance score by convolutional layers and fully-connected layers. Finally, we predict the output by a non-linear mapping based on all the relevance representations. This architecture can help to solve tasks that need reasoning on two modalities based on their relevance. We argue that the parameters trained on one task can boost the training of the other tasks that deal with multi-modality reasoning.
In this section, we first formulate the problem. Then we describe our cross-modality relevance (CMR) model for solving the problem. The architecture, loss function, and training procedure of CMR are explained in detail. We will use the VQA and NLVR tasks as showcases.
Formally, the problem is to model a mapping from a cross-modality data sample  $\mathcal{D} = \{\mathcal{D}_{\mu}\}$  to an output  $y$  in a target task, where  $\mu$  denotes the type of modality. And  $\mathcal{D}_{\mu} = \{d_1^{\mu},\dots ,d_{N^{\mu}}^{\mu}\}$  is a set of entities in the modality  $\mu$ . In visual question answering, VQA, the task is to predict an answer given two modalities, that is a textual question  $(\mathcal{D}_t)$
and a visual image  $(\mathcal{D}_v)$ . In NLVR, given a textual statement  $(\mathcal{D}_t)$  and an image  $(\mathcal{D}_v)$ , the task is to determine the correctness of the textual statement.
3.2 Representation Spaces Alignment
Single Modality Representations. For the textual modality  $\mathcal{D}_t$ , we utilize BERT (Devlin et al., 2019) as shown in the bottom-left part of Figure 1, which is a multi-layer Transformer (Vaswani et al., 2017) with three different inputs: WordPieces embeddings (Wu et al., 2016), segment embeddings, and position embeddings. We refer to all the words as the entities in the textual modality and use the BERT representations for textual single-modality representations  $\{s_1^t,\dots ,s_{N^t}^t\}$ . We assume to have  $N^t$  words as textual entities.
For visual modality  $\mathcal{D}_v$ , as shown in the top-left part of Figure 1, Faster-RCNN (Ren et al., 2015) is used to generate regions of interest (ROIs), extract dense encoding representations of the ROIs, and predict the probability of each ROI. We refer to the ROIs on images as the visual entities  $\{d_1^v,\dots ,d_{N^v}^v\}$ . We consider a fixed number,  $N^v$ , of visual entities with highest probabilities predicted by Faster-RCNN each time. The dense representation of each ROI is a local latent representation of a 2048-dimensional vector (Ren et al., 2015). To enrich the visual entity representation with the visual context, we further project the vectors with feed-forward layers and encode them by a single-modality Transformer as shown in the second column in Figure 1. The visual Transformer takes the dense representation, segment embedding, and pixel position embedding (Tan and Bansal, 2019) as input and generates the single-modality
7644
representation  $\{s_1^v,\dots ,s_{N^v}\}$ . In case there are multiple images, for example, NLVR data  $(\mathrm{NLVR}^2)$  has two images in each example, each image is encoded by the same procedure and we keep  $N^v$  visual entities per image. We refer to this as different sources of the same modality throughout the paper. We restrict all the single-modality representations to be vectors of the same dimension  $d$ . However, these original representation spaces should be aligned.
Cross-Modality Alignment. To align the single-modality representations in a uniformed representation space, we introduce a cross-modality Transformer as shown in the third column of Figure 1. All the entities are treated uniformly in the modality Transformer. Given the set of entity representations from all modalities we define the matrix with all the elements in the set  $S = \left[s_1^t,\dots ,s_{N^t}^t,s_1^v,\dots ,s_{N^v}\right]\in \mathbf{R}^{d\times (N^t +N^v)}$ . Each cross-modality self-attention calculation is computed as follows (Vaswani et al., 2017)<sup>1</sup>,
$$
\text {A t t e n t i o n} (K, Q, V) = \operatorname {s o f t m a x} \left(\frac {K ^ {\top} Q}{\sqrt {d}}\right) V, \tag {1}
$$
where in our case the key  $K$ , query  $Q$ , and value  $V$ , all are the same tensor  $S$ , and softmax  $(\cdot)$  normalizes along the columns. A cross-modality Transformer layer consists of a cross-modality self-attention representation followed by residual connection with normalization from the input representation, a feed-forward layer, and another residual connection normalization. We stack several cross-modality Transformer layers to get a uniform representation over all modalities. We refer to the resulting uniformed representations as the entity representation and denote the set of the entity representations of all the entities as  $\left\{s_1'^t,\dots ,s_{N^t}'^v,s_1'^v,\dots ,s_{N^v}'^v\right\}$ . Although the representations are still organized by their original modalities per entity, they carry the information from the interactions with the other modality and are aligned in uniform representation space. The entity representations, as the fourth column in Figure 1, alleviate the gap between representations from different modalities, as we will show in the ablation studies, and allow them to be matched in the following steps.
3.3 Entity Relevance
Relevance plays a critical role in reasoning ability, which is required in many tasks such as information retrieval, question answering, intra- and inter-modality reasoning. Relevance patterns are independent from input representation space, and can have better generalizability to unobserved data. To consider the entity relevance between two modalities  $\mathcal{D}_{\mu}$  and  $\mathcal{D}_{\nu}$ , the entity relevance representation is calculated as shown in Figure 1. Given entity representation matrices  $S^{\prime \mu} = \left[s_1^{\prime \mu},\dots ,s_{N^{\mu}}^{\prime \mu}\right]\in$ $\mathbf{R}^{d\times N^{\mu}}$  and  $S^{\prime \nu} = \left[s_1^{\prime \nu},\dots ,s_{N^{\nu}}^{\prime \nu}\right]\in \mathbf{R}^{d\times N^{\nu}}$ , the relevance representation is calculated by
$$
A ^ {\mu , \nu} = \left(S ^ {\prime \mu}\right) ^ {\top} S ^ {\prime \nu}, \tag {2a}
$$
$$
\mathrm {M} \left(\mathcal {D} _ {\mu}, \mathcal {D} _ {\nu}\right) = \mathrm {C N N} _ {\mathcal {D} _ {\mu}, \mathcal {D} _ {\nu}} \left(A ^ {\mu , \nu}\right), \tag {2b}
$$
where  $A^{\mu ,\nu}$  is the affinity matrix of the two modalities as shown in the right side of Figure 1.  $A_{ij}^{\mu ,\nu}$  is the relevance score of ith entity in  $\mathcal{D}_{\mu}$  and jth entity in  $\mathcal{D}_{\nu}$ .  $\mathrm{CNN}_{\mu ,\nu}(\cdot)$  is a CNN, corresponding to the sixth column of Figure 1, which contains several convolutional layers and fully connected layers. Each convolutional layer is followed by a max-pooling layer. Fully connected layers finally map the flatten feature maps to  $d$  -dimensional vector. We refer to  $\Phi_{\mathcal{D}_\mu ,\mathcal{D}_\nu} = \mathbf{M}(\mathcal{D}_\mu ,\mathcal{D}_\nu)$  as the entity relevance representation between  $\mu$  and  $\nu$
We compute the relevance between different modalities. For the modalities considered in this work, when there are multiple images in the visual modality, we calculate the relevance representation between them too. In particular, for VQA dataset, the above setting results in one entity relevance representation: a textual-visual entity relevance  $\Phi_{\mathcal{D}_t,\mathcal{D}_v}$ . For NLVR² dataset, there are three entity relevance representations: two textual-visual entity relevance  $\Phi_{\mathcal{D}_t,\mathcal{D}_{v_1}}$  and  $\Phi_{\mathcal{D}_t,\mathcal{D}_{v_2}}$ , and a visual-visual entity relevance  $\Phi_{\mathcal{D}_{v_1},\mathcal{D}_{v_2}}$  between two images. Entity relevance representations will be flattened and joined with other features in the next layer of the network.
3.4 Relational Relevance
We also consider the relevance beyond entities, that is, the relevance of the entities' relations. This extension allows our CMR to capture higher-order relevance patterns. We consider pair-wise non-directional relations between entities in each modality and calculate the relevance of the rela
<sup>1</sup>Please note here we keep the usual notation of the attention mechanism for this equation. The notations might have been overloaded in other parts of the paper.
7645
Figure 2: Relational Relevance is the relevance of top-K relations in terms of intra-modality relevance score and inter-modality importance.
tions across modalities. The procedure is similar to entity relevance as shown in Figure 1. We denote the relational representation as a nonlinear mapping  $\mathbf{R}^{2d}\rightarrow \mathbf{R}^d$  modeled by fullyconnected layers from the concatenation of representations of the entities in the relation  $r_{(i,j)}^{\mu} = \mathrm{MLP}_{\mu ,1}\left(\left[s_i^{\prime \mu},s_j^{\prime \mu}\right]\right)\in \mathbf{R}^d$ . Relational relevance affinity matrix can be calculated by matching the relational representation,  $\{r_{(i,j)}^{\mu},\forall i\neq j\}$ , from different modalities. However, there will be  $C_{N_{\mu}}^{2}$  possible pairs in each modality  $\mathcal{D}_{\mu}$ , most of which are irrelevant. The relational relevance representations will be sparse because of the irrelevant pairs on both sides. Computing the relevance score of all possible pairs will introduce a large number of unnecessary parameters which makes the training more difficult.
We propose to rank the relation candidates (i.e. pairs) by the intra-modality relevance score and the inter-modality importance. Then we compare the top- $K$  ranked relation candidates between two modalities as shown in Figure 2. For the intra-modality relevance score, shown in the bottom left part of the figure, we estimate a normalized score based on the relational representation by a softmax layer.
$$
U _ {(i, j)} ^ {\mu} = \frac {\exp \left(\mathrm {M L P} _ {\mu , 2} \left(r _ {(i , j)} ^ {\mu}\right)\right)}{\sum_ {k \neq l} \exp \left(\mathrm {M L P} _ {\mu , 2} \left(r _ {(k , l)} ^ {\mu}\right)\right)}. \tag {3}
$$
To evaluate the inter-modality importance of a relation candidate, which is a pair of entities in the same modality, we first compute the relevance of each entity in text with respect to the visual objects. As shown in Figure 2, we project a vector that
includes the most relevant visual object for each word, denoted this importance vector as  $v^t$ . This helps to focus on words that are grounded in the visual modality. We use the same procedure to compute the most relevant words to each visual object.
Then we calculate the relation candidates importance matrix  $V^{\mu}$  by an outer product,  $\otimes$ , of the importance vectors as follows,
$$
v _ {i} ^ {\mu} = \max  _ {j} A _ {i j} ^ {\mu , \nu}, \tag {4a}
$$
$$
V ^ {\mu} = v ^ {\mu} \otimes v ^ {\mu}, \tag {4b}
$$
where  $v_{i}^{\mu}$  is the  $i$ th scalar element in  $v^{\mu}$  that corresponds to the  $i$ th entity, and  $A^{\mu,\nu}$  is the affinity matrix calculated by Equation 2a.
Notice that the inter-modality importance  $V^{\mu}$  is symmetric. The upper triangular part of  $V^{\mu}$ , excluding the diagonal, indicates the importance of the corresponding elements with the same index in intra-modality relevance scores  $U^{\mu}$ . The ranking score for the candidates is the combination (here the product) of the two scores  $W_{(i,j)}^{\mu} = U_{(i,j)}^{\mu} \times V_{ij}^{\mu}$ . We select the set of top- $K$  ranked candidate relations  $\mathcal{K}_{\mu} = \{\kappa_1, \kappa_2, \dots, \kappa_K\}$ . We reorganize the representation of the top- $K$  relations as  $R^{\mu} = [r_{\kappa_1}^{\mu}, \dots, r_{\kappa_K}^{\mu}] \in \mathbf{R}^{d \times K}$ . The relational relevance representation between  $\mathcal{K}_{\mu}$  and  $\mathcal{K}_{\nu}$  can be calculated similar to the entity relevance representations as shown in Figure 1.
$$
\mathbf {M} \left(\mathcal {K} _ {\mu}, \mathcal {K} _ {\nu}\right) = \operatorname {C N N} _ {\mathcal {K} _ {\mu}, \mathcal {K} _ {\nu}} \left(\left(R ^ {\mu}\right) ^ {\top} R ^ {\nu}\right). \tag {5}
$$
$\mathbf{M}(\mathcal{K}_{\mu},\mathcal{K}_{\nu})$  has its own parameters which results in a  $d$  -dimensional feature space  $\Phi_{\mathcal{K}_\mu ,\mathcal{K}_\nu}$
In particular, for VQA task, the above setting results in one relational relevance representation: a textual-visual relevance  $\mathbf{M}(\mathcal{K}_t,\mathcal{K}_v)$ . For NLVR task, there are three entity relevance representations: two textual-visual relational relevance  $\mathbf{M}(\mathcal{K}_t,\mathcal{K}_{v_1})$  and  $\mathbf{M}(\mathcal{K}_t,\mathcal{K}_{v_2})$ , and a visual-visual relational relevance  $\mathbf{M}(\mathcal{K}_{v_1},\mathcal{K}_{v_2})$  between two images. Relational relevance representations will be flattened and joined with other features in the next layers of the network.
After acquiring all the entity and relational relevance representations, namely  $\Phi_{\mathcal{D}_{\mu},\mathcal{D}_{\nu}}$  and  $\Phi_{\mathcal{K}_{\mu},\mathcal{K}_{\nu}}$  we concatenate them and use the result as the final feature  $\Phi = \left[\Phi_{\mathcal{D}_{\mu},\mathcal{D}_{\nu}},\dots ,\Phi_{\mathcal{K}_{\mu},\mathcal{K}_{\nu}},\dots \right]$ . A task-specific classifier  $\mathrm{MLP}_{\Phi}(\Phi)$  predicts the output of the target task as shown in the right-most column in Figure 1.
7646
End-to-end Training. CMR can be considered as an end-to-end relevance representation extractor. We simply predict the output  $y$  from a specific task with the final feature  $\Phi$  with a differentiable regression or classification function. The gradient of the loss function is backpropagated to all the components in CMR to penalize the prediction and adjust the parameters. We freeze the parameters of the basic feature extractors, namely BERT for textual modality and Faster-RCNN for visual modality. The parameters of the following parts will be updated by gradient descent: single modality Transformers (except BERT), the cross-modality Transformers,  $\mathrm{CNN}_{\mathcal{D}_{\mu},\mathcal{D}_{\nu}}(\cdot)$ ,  $\mathrm{CNN}_{\mathcal{K}_{\mu},\mathcal{K}_{\nu}}(\cdot)$ ,  $\mathrm{MLP}_{\mu,1}(\cdot)$ ,  $\mathrm{MLP}_{\mu,2}(\cdot)$  for all modalities and modality pairs, and the task-specific classifier  $\mathrm{MLP}_{\Phi}(\Phi)$ .
The VQA task can be formulated as a multi-class classification that chooses a word to answer the question. We apply a softmax classifier on  $\Phi$  and penalize with the cross-entropy loss. For  $\mathrm{NLVR}^2$  dataset, the task is binary classification that determines whether the statement is correct regarding the images. We apply a logistic regression on  $\Phi$  and penalize with the cross-entropy loss.