SlowGuess/ACL_2020_OCR huggingface dataset:
Jianfei Yu $^{1}$ , Jing Jiang $^{2}$ , Li Yang $^{3}$ , and Rui Xia $^{1,*}$
<sup>1</sup> School of Artificial Intelligence, Nanjing University of Science & Technology, China
$^{2}$  School of Information Systems, Singapore Management University, Singapore
$^{3}$  DBS Bank, Singapore
{jfyu, rxia}@njust.edu.cn, jingjiang@smu.edu.sg, liyang@db.com
In this paper, we study Multimodal Named Entity Recognition (MNER) for social media posts. Existing approaches for MNER mainly suffer from two drawbacks: (1) despite generating word-aware visual representations, their word representations are insensitive to the visual context; (2) most of them ignore the bias brought by the visual context. To tackle the first issue, we propose a multimodal interaction module to obtain both image-aware word representations and word-aware visual representations. To alleviate the visual bias, we further propose to leverage purely text-based entity span detection as an auxiliary module, and design a Unified Multimodal Transformer to guide the final predictions with the entity span predictions. Experiments show that our unified approach achieves the new state-of-the-art performance on two benchmark datasets.
Recent years have witnessed the explosive growth of user-generated contents on social media platforms such as Twitter. While empowering users with rich information, the flourish of social media also solicits the emerging need of automatically extracting important information from these massive unstructured contents. As a crucial component of many information extraction tasks, named entity recognition (NER) aims to discover named entities in free text and classify them into pre-defined types, such as person (PER), location (LOC) and organization (ORG). Given its importance, NER has attracted much attention in the research community (Yadav and Bethard, 2018).
Although many methods coupled with either discrete shallow features (Zhou and Su, 2002; Finkel et al., 2005; Torisawa et al., 2007) or continuous deep features (Lample et al., 2016; Ma and Hovy,
(a). [Kevin Durant PER] enters [Oracle Arena LOC] wearing off — White x [Jordan MISC]
Figure 1: Two examples for Multimodal Named Entity Recognition (MNER). Named entities and their entity types are highlighted.
(b).Vote for [King of the Jungle MISC] - [Kian PER] or [David PER] ?
2016) have shown success in identifying entities in formal newswire text, most of them perform poorly on informal social media text (e.g., tweets) due to its short length and noisiness. To adapt existing NER models to social media, various methods have been proposed to incorporate many tweet-specific features (Ritter et al., 2011; Li et al., 2012, 2014; Limsopatham and Collier, 2016). More recently, as social media posts become increasingly multimodal, several studies proposed to exploit useful visual information to improve the performance of NER (Moon et al., 2018; Zhang et al., 2018; Lu et al., 2018).
In this work, following the recent trend, we focus on multimodal named entity recognition (MNER) for social media posts, where the goal is to detect named entities and identify their entity types given a {sentence, image} pair. For example, in Fig. 1.a, it is expected to recognize that Kevin Durant, Oracle Arena, and Jordan belong to the category of person names (i.e., PER), place names (i.e., LOC), and other names (i.e., MISC), respectively.
While previous work has shown success of fusing visual information into NER (Moon et al., 2018; Zhang et al., 2018; Lu et al., 2018), they still suffer from several limitations: (1) The first obstacle
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3342-3352 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
lies in the non-contextualized word representations, where each word is represented by the same vector, regardless of the context it occurs in. However, the meanings of many polysemous entities in social media posts often rely on its context words. Take Fig. 1.a as an example, without the context words wearing off, it is hard to figure out whether Jordan refers to a shoe brand or a person. (2) Although most existing methods focus on modeling inter-modal interactions to obtain word-aware visual representations, the word representations in their final hidden layer are still based on the textual context, which are insensitive to the visual context. Intuitively, the associated image often provides more context to resolve polysemous entities, and should contribute to the final word representations (e.g., in Fig. 1.b, the image can supervise the final word representations of Kian and David to be closer to persons than animals). (3) Most previous approaches largely ignore the bias of incorporating visual information. Actually, in most social media posts, the associated image tends to highlight only one or two entities in the sentence, without mentioning the other entities. In these cases, directly integrating visual information will inevitably lead the model to better recognize entities highlighted by images, but fail to identify the other entities (e.g., Oracle Arena and King of the Jungle in Fig. 1).
To address these limitations, we resort to existing pre-trained contextualized word representations, and propose a unified multimodal architecture based on Transformer (Vaswani et al., 2017), which can effectively capture inter-modality interactions and alleviate the visual bias. Specifically, we first adopt a recently pre-trained contextualized representation model (Devlin et al., 2018) as our sentence encoder, whose multi-head self-attention mechanism can guide each word to capture the semantic and syntactic dependency upon its context. Second, to better capture the implicit alignments between words and images, we propose a multimodal interaction (MMI) module, which essentially couples the standard Transformer layer with cross-modal attention mechanism to produce an image-aware word representation and a word-aware visual representation for each input word, respectively. Finally, to largely eliminate the bias of the visual context, we propose to leverage text-based entity span detection as an auxiliary task, and design a unified neural architecture based on Transformer. In particular, a conversion matrix is
designed to construct the correspondence between the auxiliary and the main tasks, so that the entity span information can be fully utilized to guide the final MNER predictions.
Experimental results show that our Unified Multimodal Transformer (UMT) brings consistent performance gains over several highly competitive unimodal and multimodal methods, and outperforms the state-of-the-art by a relative improvement of  $3.7\%$  and  $3.8\%$  on two benchmarks, respectively.
- We propose a Multimodal Transformer model for the task of MNER, which empowers Transformer with a multimodal interaction module to capture the inter-modality dynamics between words and images. To the best of our knowledge, this is the first work to apply Transformer to MNER.
- Based on the above Multimodal Transformer, we further design a unified architecture to incorporate a text-based entity span detection module, aiming to alleviate the bias of the visual context in MNER with the guidance of entity span predictions from this auxiliary module.
In this section, we first formulate the MNER task, and give an overview of our method. We then delve into the details of each component in our model.
Task Formulation: Given a sentence  $S$  and its associated image  $V$  as input, the goal of MNER is to extract a set of entities from  $S$ , and classify each extracted entity into one of the pre-defined types.
As with most existing work on MNER, we formulate the task as a sequence labeling problem. Let  $S = (s_1, s_2, \ldots, s_n)$  denote a sequence of input words, and  $\mathbf{y} = (y_1, y_2, \ldots, y_n)$  be the corresponding label sequence, where  $y_i \in \mathcal{V}$  and  $\mathcal{V}$  is the pre-defined label set with the BIO2 tagging schema (Sang and Veenstra, 1999).
Fig. 2.a illustrates the overall architecture of our Unified Multimodal Transformer, which contains three main components: (1) representation learning for unimodal input; (2) a Multimodal Transformer for MNER; and (3) a unified architecture with auxiliary entity span detection (ESD) module.
Figure 2: (a). Overall Architecture of Our Unified Multimodal Transformer.
(b). Multimodal Interaction (MMI) Module.
As shown at the bottom of Fig. 2.a, we first extract contextualized word representations and visual block representations from the input sentence and the input image, respectively.
The right part of Fig. 2.a illustrates our Multimodal Transformer model for MNER. Specifically, a Transformer layer is first employed to derive each word's textual hidden representation. Next, a multimodal interaction (MMI) module is devised to fully capture the inter-modality dynamics between the textual hidden representations and the visual block representations. The hidden representations from MMI are then fed to a conditional random field (CRF) layer to produce the label for each word.
To alleviate the visual bias in MNER, we further stack a purely text-based ESD module in the left part of Fig. 2.a, where we feed its hidden representations to another CRF layer to predict each word's entity span label. More importantly, to utilize this for our main MNER task, we design a conversion matrix to encode the dependency relations between corresponding labels from ESD to MNER, so that the entity span predictions from ESD can be integrated to get the final MNER label for each word.
2.2 Unimodal Input Representations
Word Representations: Due to the capability of giving different representations for the same word in different contexts, we employ the recent contextualized representations from BERT (Devlin et al.,
2018) as our sentence encoder. Following Devlin et al. (2018), each input sentence is preprocessed by inserting two special tokens, i.e., appending [CLS] to the beginning and [SEP] to the end, respectively. Formally, let  $S' = (s_0, s_1, \ldots, s_{n+1})$  be the modified input sentence, where  $s_0$  and  $s_{n+1}$  denote the two inserted tokens. Let  $\mathbf{X} = (\mathbf{x}_0, \mathbf{x}_1, \ldots, \mathbf{x}_{n+1})$  be the word representations of  $S'$ , where  $\mathbf{x}_i$  is the sum of word, segment, and position embeddings for each token  $s_i$ . As shown in the bottom left of Fig. 2.a,  $\mathbf{X}$  is then fed to the BERT encoder to obtain  $\mathbf{C} = (\mathbf{c}_0, \mathbf{c}_1, \ldots, \mathbf{c}_{n+1})$ , where  $\mathbf{c}_i \in \mathbb{R}^d$  is the generated contextualized representation for  $\mathbf{x}_i$ .
Visual Representations: As one of the state-of-the-art CNN models for image recognition, Residual Network (ResNet) (He et al., 2016) has shown its capability of extracting meaningful feature representations of the input image in its deep layers. We therefore keep the output from the last convolutional layer in a pretrained 152-layer ResNet to represent each image, which essentially splits each input image into  $7 \times 7 = 49$  visual blocks with the same size and represents each block with a 2048-dimensional vector. Specifically, given an input image  $V$ , we first resize it to  $224 \times 224$  pixels, and obtain its visual representations from ResNet, denoted as  $\mathbf{U} = (\mathbf{u}_1, \mathbf{u}_2, \dots, \mathbf{u}_{49})$ , where  $\mathbf{u}_i$  is the 2048-dimensional vector representation for the  $i$ -th visual block. To project the visual representations into the same space of the word representations,
we further convert  $\mathbf{U}$  with a linear transformation:  $\mathbf{V} = \mathbf{W}_u^\top \mathbf{U}$ , where  $\mathbf{W}_u\in \mathbb{R}^{2048\times d}$  is the weight matrix. As shown in the bottom right of Fig. 2.a,  $\mathbf{V} = (\mathbf{v}_1,\mathbf{v}_2,\dots ,\mathbf{v}_{49})$  is the visual representations generated from ResNet.
2.3 Multimodal Transformer for MNER
In this subsection, we present our proposed Multimodal Transformer for MNER.
As illustrated on the right of Fig. 2.a, we first add a standard Transformer layer over  $\mathbf{C}$  to obtain each word's textual hidden representation:  $\mathbf{R} = (\mathbf{r}_0,\mathbf{r}_1,\dots ,\mathbf{r}_{n + 1})$  where  $\mathbf{r}_i\in \mathbb{R}^d$  denotes the generated hidden representation for  $\mathbf{x}_i$ .
Motivation: While the above Transformer layer can capture which context words are more relevant to the prediction of an input word  $\mathbf{x}_i$ , they fail to consider the associated visual context. On the one hand, due to the short length of textual contents on social media, the additional visual context may guide each word to learn better word representations. On the other hand, since each visual block is often closely related to several input words, incorporating the visual block representation can potentially make the prediction of its related words more accurately. Inspired by these observations, we propose a multimodal interaction (MMI) module to learn an image-aware word representation and a word-aware visual representation for each word.
2.3.1 Image-Aware Word Representation
Cross-Modal Transformer (CMT) Layer: As shown on the left of Fig. 2.b, to learn better word representations with the guidance of associated images, we first employ an  $m$ -head cross-modal attention mechanism (Tsai et al., 2019), by treating  $\mathbf{V} \in \mathbb{R}^{d \times 49}$  as queries, and  $\mathbf{R} \in \mathbb{R}^{d \times (n + 1)}$  as keys and values:
$$
\mathrm {C A} _ {i} (\mathbf {V}, \mathbf {R}) = \operatorname {s o f t m a x} \left(\frac {\left[ \mathbf {W} _ {\mathbf {q} _ {\mathrm {i}}} \mathbf {V} \right] ^ {\top} \left[ \mathbf {W} _ {\mathbf {k} _ {\mathrm {i}}} \mathbf {R} \right]}{\sqrt {d / m}}\right) \left[ \mathbf {W} _ {\mathbf {v} _ {\mathrm {i}}} \mathbf {R} \right] ^ {\top};
$$
$$
\mathbf {M H - C A} (\mathbf {V}, \mathbf {R}) = \mathbf {W} ^ {\prime} [ \mathrm {C A} _ {1} (\mathbf {V}, \mathbf {R}), \dots , \mathrm {C A} _ {m} (\mathbf {V}, \mathbf {R}) ] ^ {\top},
$$
where  $\mathrm{CA}_i$  refers to the  $i$ -th head of cross-modal attention,  $\{\mathbf{W}_{\mathbf{q}_i}, \mathbf{W}_{\mathbf{k}_i}, \mathbf{W}_{\mathbf{v}_i}\} \in \mathbb{R}^{d / m \times d}$ , and  $\mathbf{W}' \in \mathbb{R}^{d \times d}$  denote the weight matrices for the query, key, value, and multi-head attention, respectively. Next, we stack another three sub-layers on top:
$$
\widetilde {\mathbf {P}} = \operatorname {L N} (\mathbf {V} + \operatorname {M H - C A} (\mathbf {V}, \mathbf {R})); \tag {1}
$$
$$
\mathbf {P} = \mathrm {L N} (\widetilde {\mathbf {P}} + \operatorname {F F N} (\widetilde {\mathbf {P}})), \tag {2}
$$
where FFN is the feed-forward network (Vaswani et al., 2017), LN is the layer normalization (Ba et al., 2016), and  $\mathbf{P} = (\mathbf{p}_1, \mathbf{p}_2, \dots, \mathbf{p}_{49})$  is the output representations of the CMT layer.
Coupled CMT Layer: However, since the visual representations are treated as queries in the above CMT layer, each generated vector  $\mathbf{p}_i$  is corresponding to the  $i$ -th visual block instead of the  $i$ -th input word. Ideally, the image-aware word representation should be corresponding to each word.
To address this, we propose to couple  $\mathbf{P}$  with another CMT layer, which treats the textual representations  $\mathbf{R}$  as queries, and  $\mathbf{P}$  as keys and values. As shown in the top left of Fig. 2.a, this coupled CMT layer generates the final image-aware word representations, denoted by  $\mathbf{A} = (\mathbf{a}_0, \mathbf{a}_1, \dots, \mathbf{a}_{n+1})$ .
2.3.2 Word-Aware Visual Representation
To obtain a visual representation for each word, it is necessary to align each word with its closely related visual blocks, i.e., assigning high/low attention weights to its related/unrelated visual blocks. Hence, as shown in the right part of Fig. 2.b, we use a CMT layer by treating  $\mathbf{R}$  as queries and  $\mathbf{V}$  as keys and values, which can be considered as a symmetric version of the left CMT layer. Finally, it generates the word-aware visual representations, denoted by  $\mathbf{Q} = (\mathbf{q}_0,\mathbf{q}_1,\dots ,\mathbf{q}_{n + 1})$
Visual Gate: As pointed out in some previous studies (Zhang et al., 2018; Lu et al., 2018), it is unreasonable to align many function words such as the, of, and well with any visual block. Therefore, it is important to incorporate a visual gate to dynamically control the contribution of visual features. Following the practice in previous work, we design a visual gate by combining the information from the above word representations  $\mathbf{A}$  and visual representations  $\mathbf{Q}$  as follows:
$$
\mathbf {g} = \sigma \left(\mathbf {W} _ {a} ^ {\top} \mathbf {A} + \mathbf {W} _ {q} ^ {\top} \mathbf {Q}\right), \tag {3}
$$
where  $\{\mathbf{W}_a,\mathbf{W}_q\} \in \mathbb{R}^{d\times d}$  are weight matrices, and  $\sigma$  is the element-wise sigmoid function. Based on the gate output, we can obtain the final word-aware visual representations as  $\mathbf{B} = \mathbf{g}\cdot \mathbf{Q}$ .
2.3.3 CRF Layer
To integrate the word and the visual representations, we concatenate  $\mathbf{A}$  and  $\mathbf{B}$  to obtain the final hidden representations  $\mathbf{H} = (\mathbf{h}_0,\mathbf{h}_1,\dots ,\mathbf{h}_{n + 1})$  , where  $\mathbf{h}_i\in \mathbb{R}^{2d}$  . Following Lample et al. (2016), we then feed  $\mathbf{H}$  to a standard CRF layer, which defines the
<sup>1</sup>Bias terms are omitted to avoid confusion in this paper.
probability of the label sequence  $\mathbf{y}$  given the input sentence  $S$  and its associated image  $V$ :
$$
P (\mathbf {y} | S, V) = \frac {\exp (s c o r e (\mathbf {H} , \mathbf {y}))}{\sum_ {\mathbf {y} ^ {\prime}} \exp (s c o r e (\mathbf {H} , \mathbf {y} ^ {\prime}))}; \tag {4}
$$
$$
\operatorname {s c o r e} (\mathbf {H}, \mathbf {y}) = \sum_ {i = 0} ^ {n} T _ {y _ {i}, y _ {i + 1}} + \sum_ {i = 1} ^ {n} E _ {\mathbf {h} _ {i}, y _ {i}}; \tag {5}
$$
$$
E _ {\mathbf {h} _ {i}, y _ {i}} = \mathbf {w} _ {\text {M N E R}} ^ {y _ {i}} \cdot \mathbf {h} _ {i}, \tag {6}
$$
where  $T_{y_i,y_{i + 1}}$  is the transition score from the label  $y_{i}$  to the label  $y_{i + 1}$ ,  $E_{\mathbf{h}_{i},y_{i}}$  is the emission score of the label  $y_{i}$  for the  $i$ -th word, and  $\mathbf{w}_{\mathrm{MNER}}^{y_i} \in \mathbb{R}^{2d}$  is the weight parameter specific to  $y_{i}$ .
2.4 Unified Multimodal Transformer
Motivation: Since the Multimodal Transformer presented above mainly focuses on modeling the interactions between text and images, it may lead the learnt model to overemphasize the entities highlighted by the image but ignore the remaining entities. To alleviate the bias, we propose to leverage text-based entity span detection (ESD) as an auxiliary task based on the following observation. As ResNet is pre-trained on ImageNet (Deng et al., 2009) for the image recognition task, its high-level representations are closely relevant to the final predictions, i.e., the types of contained objects. This indicates that the visual representations from ResNet should be quite useful for identifying types of the detected entities, but are not necessarily relevant to detecting entity spans in the sentence. Therefore, we use purely text-based ESD to guide the final predictions for our main MNER task.
Auxiliary Entity Span Detection Module: Formally, we model ESD as another sequence labeling task, and use  $\mathbf{z} = (z_1,\dots ,z_n)$  to denote the sequence of labels, where  $z_{i}\in \mathcal{Z}$  and  $\mathcal{Z} = \{\mathrm{B},\mathrm{I},\mathrm{O}\}$ .
As shown in the left part of Fig. 2.a, we employ another Transformer layer to obtain its specific hidden representations as  $\mathbf{T} = (\mathbf{t}_0,\mathbf{t}_1,\dots ,\mathbf{t}_{n + 1})$  followed by feeding it to a CRF layer to predict the probability of the label sequence  $\mathbf{z}$  given  $S$ :
$$
P (\mathbf {z} | S) = \frac {\exp (\sum_ {i = 0} ^ {n} T _ {z _ {i} , z _ {i + 1}} + \sum_ {i = 1} ^ {n} \mathbf {w} _ {\mathrm {E S D}} ^ {z _ {i}} \cdot \mathbf {t} _ {i})}{\sum_ {\mathbf {z} ^ {\prime}} \exp (\sum_ {i = 0} ^ {n} T _ {z _ {i} ^ {\prime} , z _ {i + 1} ^ {\prime}} + \sum_ {i = 1} ^ {n} \mathbf {w} _ {\mathrm {E S D}} ^ {z _ {i} ^ {\prime}} \cdot \mathbf {t} _ {i})},
$$
where  $\mathbf{w}_{\mathrm{ESD}}^{z_i} \in \mathbb{R}^d$  is the parameter specific to  $z_{i}$ .
Conversion Matrix: Although ESD is modeled as an auxiliary task separated from MNER, the two tasks are highly correlated since each ESD label should be only corresponding to a subset of labels in MNER. For example, given the sentence in Fig. 2.a, if the first token is predicted to be the
Entity Type	TWITTER-2015	TWITTER-2017
Train	Dev	Test	Train	Dev	Test
Person	2217	552	1816	2943	626	621
Location	2091	522	1697	731	173	178
Organization	928	247	839	1674	375	395
Miscellaneous	940	225	726	701	150	157
Total	6176	1546	5078	6049	1324	1351
Num of Tweets	4000	1000	3257	3373	723	723
Table 1: The basic statistics of our two Twitter datasets.
beginning of an entity in ESD (i.e., have the label  $B$ ), it should be also the beginning of a typed entity in MNER (e.g., have the label  $B-PER$ ).
To encode such inter-task correspondence, we propose to use a conversion matrix  $\mathbf{W}^c\in \mathbb{R}^{|\mathcal{Z}|\times |\mathcal{Y}|}$  where each element  $\mathbf{W}_{j,k}^c$  defines the conversion probability from  $\mathcal{Z}_j$  to  $\mathcal{V}_k$ . Since we have some prior knowledge (e.g., the label  $B$  can only convert to a label subset  $\{B\text{-PER}, B\text{-LOC}, B\text{-ORG}, B\text{-MISC}\}$ ), we initialize  $\mathbf{W}^c$  as follows: if  $\mathcal{Z}_j$  is not corresponding to  $\mathcal{V}_k$ ,  $\mathbf{W}_{j,k}^c$  is set to 0; otherwise,  $\mathbf{W}_{j,k}^c$  is set to  $\frac{1}{|\mathbf{C}_j|}$ , where  $\mathbf{C}_j$  denotes a subset of  $\mathcal{V}$  that is corresponding to  $\mathcal{Z}_j$ .
Modified CRF Layer for MNER: After obtaining the conversion matrix, we further propose to fully leverage the text-based entity span predictions to guide the final predictions of MNER. Specifically, we modify the CRF layer for MNER by incorporating the entity span information from ESD into the emission score defined in Eqn. (6):
$$
E _ {\mathbf {h} _ {i}, y _ {i}} = \mathbf {w} _ {\text {M N E R}} ^ {y _ {i}} \cdot \mathbf {h} _ {i} + \mathbf {w} _ {\text {E S D}} ^ {z _ {i}} \cdot \mathbf {t} _ {i} \cdot \mathbf {W} _ {z _ {i}, y _ {i}} ^ {c}. \tag {7}
$$
2.5 Model Training
Given a set of manually labeled training samples  $D = \{S^j, V^j, \mathbf{y}^j, \mathbf{z}^j\}_{j=1}^N$ , our overall training objective function is a weighted sum of the sentence-level negative log-likelihood losses for our main MNER task and the auxiliary ESD task:
$$
\mathcal {L} = - \frac {1}{| D |} \sum_ {j = 1} ^ {N} \left(\log P \left(\mathbf {y} ^ {j} \mid S ^ {j}, V ^ {j}\right) + \lambda \log P \left(\mathbf {z} ^ {j} \mid S ^ {j}\right)\right),
$$
where  $\lambda$  is a hyperparameter to control the contribution of the auxiliary ESD module.
We conduct experiments on two multimodal NER datasets, comparing our Unified Multimodal Transformer (UMT) with a number of unimodal and multimodal approaches.
2We obtain  $\mathbf{z}^j$  by removing the type information in  $\mathbf{y}^j$ .