Venue: ACL 2020

Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer

Jianfei Yu\({}^{1}\), Jing Jiang\({}^{2}\), Li Yang\({}^{3}\), and Rui Xia\({}^{1,*}\)

\({}^{1}\) School of Artificial Intelligence, Nanjing University of Science & Technology, China

\({}^{2}\) School of Information Systems, Singapore Management University, Singapore

\({}^{3}\) DBS Bank, Singapore

{jfyu, rxia}@njust.edu.cn, jingjiang@smu.edu.sg, liyang@dbs.com

Corresponding author.

###### Abstract

In this paper, we study Multimodal Named Entity Recognition (MNER) for social media posts. Existing approaches for MNER mainly suffer from two drawbacks: (1) despite generating word-aware visual representations, their word representations are insensitive to the visual context; (2) most of them ignore the bias brought by the visual context. To tackle the first issue, we propose a multimodal interaction module to obtain both image-aware word representations and word-aware visual representations. To alleviate the visual bias, we further propose to leverage purely text-based entity span detection as an auxiliary module, and design a Unified Multimodal Transformer to guide the final predictions with the entity span predictions. Experiments show that our unified approach achieves the new state-of-the-art performance on two benchmark datasets.

## 1 Introduction

Recent years have witnessed the explosive growth of user-generated contents on social media platforms such as Twitter. While empowering users with rich information, the flourish of social media also solicits the emerging need of automatically extracting important information from these massive unstructured contents. As a crucial component of many information extraction tasks, named entity recognition (NER) aims to discover named entities in free text and classify them into pre-defined types, such as person (_PER_), location (_LOC_) and organization (_ORG_). Given its importance, NER has attracted much attention in the research community Yadav and Bethard (2018).

Although many methods coupled with either discrete shallow features Zhou and Su (2002); Finkel et al. (2005); Torisawa et al. (2007) or continuous deep features Lample et al. (2016); Ma and Hovy (2016) have shown success in identifying entities in formal newswire text, most of them perform poorly on informal social media text (e.g., tweets) due to its short length and noisiness. To adapt existing NER models to social media, various methods have been proposed to incorporate many tweet-specific features Ritter et al. (2011); Li et al. (2012, 2014); Limspatham and Collier (2016). More recently, as social media posts become increasingly multi-modal, several studies proposed to exploit useful visual information to improve the performance of NER Moon et al. (2018); Zhang et al. (2018); Lu et al. (2018).

In this work, following the recent trend, we focus on multimodal named entity recognition (MNER) for social media posts, where the goal is to detect named entities and identify their entity types given a {sentence, image} pair. For example, in Fig. 0(a), it is expected to recognize that _Kevin Durant_, _Oracle Arena_, and _Jordan_ belong to the category of person names (i.e., _PER_), place names (i.e., _LOC_), and other names (i.e., _MISC_), respectively.

While previous work has shown success of fusing visual information into NER Moon et al. (2018); Zhang et al. (2018); Lu et al. (2018), they still suffer from several limitations: (1) The first obstacle

Figure 1: Two examples for Multimodal Named Entity Recognition (MNER). Named entities and their entity types are highlighted.

 

lies in the non-contextualized word representations, where each word is represented by the same vector, regardless of the context it occurs in. However, the meanings of many polysemous entities in social media posts often rely on its context words. Take Fig. 1.a as an example, without the context words _wearing off_, it is hard to figure out whether _Jordan_ refers to a shoe brand or a person. (2) Although most existing methods focus on modeling inter-modal interactions to obtain word-aware visual representations, the word representations in their final hidden layer are still based on the textual context, which are insensitive to the visual context. Intuitively, the associated image often provides more context to resolve polysemous entities, and should contribute to the final word representations (e.g., in Fig. 1.b, the image can supervise the final word representations of _Kian_ and _David_ to be closer to persons than animals). (3) Most previous approaches largely ignore the bias of incorporating visual information. Actually, in most social media posts, the associated image tends to highlight only one or two entities in the sentence, without mentioning the other entities. In these cases, directly integrating visual information will inevitably lead the model to better recognize entities highlighted by images, but fail to identify the other entities (e.g., _Oracle Arena_ and _King of the Jungle_ in Fig. 1).

To address these limitations, we resort to existing pre-trained contextualized word representations, and propose a unified multimodal architecture based on Transformer Vaswani et al. (2017), which can effectively capture inter-modality interactions and alleviate the visual bias. Specifically, we first adopt a recently pre-trained contextualized representation model Devlin et al. (2018) as our sentence encoder, whose multi-head self-attention mechanism can guide each word to capture the semantic and syntactic dependency upon its context. Second, to better capture the implicit alignments between words and images, we propose a multimodal interaction (MMI) module, which essentially couples the standard Transformer layer with cross-modal attention mechanism to produce an image-aware word representation and a word-aware visual representation for each input word, respectively. Finally, to largely eliminate the bias of the visual context, we propose to leverage text-based entity span detection as an auxiliary task, and design a unified neural architecture based on Transformer. In particular, a conversion matrix is designed to construct the correspondence between the auxiliary and the main tasks, so that the entity span information can be fully utilized to guide the final MNER predictions.

Experimental results show that our Unified Multimodal Transformer (UMT) brings consistent performance gains over several highly competitive unimodal and multimodal methods, and outperforms the state-of-the-art by a relative improvement of 3.7% and 3.8% on two benchmarks, respectively.

The main contributions of this paper can be summarized as follows:

* We propose a Multimodal Transformer model for the task of MNER, which empowers Transformer with a multimodal interaction module to capture the inter-modality dynamics between words and images. To the best of our knowledge, this is the first work to apply Transformer to MNER.
* Based on the above Multimodal Transformer, we further design a unified architecture to incorporate a text-based entity span detection module, aiming to alleviate the bias of the visual context in MNER with the guidance of entity span predictions from this auxiliary module.

## 2 Methodology

In this section, we first formulate the MNER task, and give an overview of our method. We then delve into the details of each component in our model.

**Task Formulation:** Given a sentence \(S\) and its associated image \(V\) as input, the goal of MNER is to extract a set of entities from \(S\), and classify each extracted entity into one of the pre-defined types.

As with most existing work on MNER, we formulate the task as a sequence labeling problem. Let \(S=(s_{1},s_{2},\ldots,s_{n})\) denote a sequence of input words, and \(\textbf{y}=(y_{1},y_{2},\ldots,y_{n})\) be the corresponding label sequence, where \(y_{i}\in\mathcal{Y}\) and \(\mathcal{Y}\) is the pre-defined label set with the _BIO2_ tagging schema Sang and Veenstra (1999).

### Overall Architecture

Fig. 2.a illustrates the overall architecture of our Unified Multimodal Transformer, which contains three main components: (1) representation learning for unimodal input; (2) a Multimodal Transformer for MNER; and (3) a unified architecture with auxiliary entity span detection (ESD) module 

As shown at the bottom of Fig. 2.a, we first extract contextualized word representations and visual block representations from the input sentence and the input image, respectively.

The right part of Fig. 2.a illustrates our Multimodal Transformer model for MNER. Specifically, a Transformer layer is first employed to derive each word's textual hidden representation. Next, a multimodal interaction (MMI) module is devised to fully capture the inter-modality dynamics between the textual hidden representations and the visual block representations. The hidden representations from MMI are then fed to a conditional random field (CRF) layer to produce the label for each word.

To alleviate the visual bias in MNER, we further stack a purely text-based ESD module in the left part of Fig. 2.a, where we feed its hidden representations to another CRF layer to predict each word's entity span label. More importantly, to utilize this for our main MNER task, we design a conversion matrix to encode the dependency relations between corresponding labels from ESD to MNER, so that the entity span predictions from ESD can be integrated to get the final MNER label for each word.

### Unimodal Input Representations

**Word Representations**: Due to the capability of giving different representations for the same word in different contexts, we employ the recent contextualized representations from BERT Devlin et al. (2018) as our sentence encoder. Following Devlin et al. (2018), each input sentence is preprocessed by inserting two special tokens, i.e., appending [CLS] to the beginning and [SEP] to the end, respectively. Formally, let \(S^{\prime}=(s_{0},s_{1},\dots,s_{n+1})\) be the modified input sentence, where \(s_{0}\) and \(s_{n+1}\) denote the two inserted tokens. Let \(\mathbf{X}=(\mathbf{x}_{0},\mathbf{x}_{1},\dots,\mathbf{x}_{n+1})\) be the word representations of \(S^{\prime}\), where \(\mathbf{x}_{i}\) is the sum of word, segment, and position embeddings for each token \(s_{i}\). As shown in the bottom left of Fig. 2.a, \(\mathbf{X}\) is then fed to the BERT encoder to obtain \(\mathbf{C}=(\mathbf{c}_{0},\mathbf{c}_{1},\dots,\mathbf{c}_{n+1})\), where \(\mathbf{c}_{i}\in\mathbb{R}^{d}\) is the generated contextualized representation for \(\mathbf{x}_{i}\).

**Visual Representations**: As one of the state-of-the-art CNN models for image recognition, Residual Network (ResNet) He et al. (2016) has shown its capability of extracting meaningful feature representations of the input image in its deep layers. We therefore keep the output from the last convolutional layer in a pretrained 152-layer ResNet to represent each image, which essentially splits each input image into 7\(\times\)7=49 visual blocks with the same size and represents each block with a 2048-dimensional vector. Specifically, given an input image \(V\), we first resize it to 224\(\times\)224 pixels, and obtain its visual representations from ResNet, denoted as \(\mathbf{U}=(\mathbf{u}_{1},\mathbf{u}_{2},\dots,\mathbf{u}_{49})\), where \(\mathbf{u}_{i}\) is the 2048-dimensional vector representation for the \(i\)-th visual block. To project the visual representations into the same space of the word representations,

Figure 2: (a). Overall Architecture of Our Unified Multimodal Transformer.

 

we further convert \({\bf U}\) with a linear transformation: \({\bf V}={\bf W}_{u}^{\top}{\bf U}\), where \({\bf W}_{u}\in\mathbb{R}^{2048\times d}\) is the weight matrix1. As shown in the bottom right of Fig. 2.a, \({\bf V}=({\bf v}_{1},{\bf v}_{2},\ldots,{\bf v}_{49})\) is the visual representations generated from ResNet.

Footnote 1: Bias terms are omitted to avoid confusion in this paper.

### Multimodal Transformer for MNER

In this subsection, we present our proposed Multimodal Transformer for MNER.

As illustrated on the right of Fig. 2.a, we first add a standard Transformer layer over \({\bf C}\) to obtain each word's textual hidden representation: \({\bf R}=({\bf r}_{0},{\bf r}_{1},\dots,{\bf r}_{n+1})\), where \({\bf r}_{i}\in\mathbb{R}^{d}\) denotes the generated hidden representation for \({\bf x}_{i}\).

**Motivation:** While the above Transformer layer can capture which context words are more relevant to the prediction of an input word \({\bf x}_{i}\), they fail to consider the associated visual context. On the one hand, due to the short length of textual contents on social media, the additional visual context may guide each word to learn better word representations. On the other hand, since each visual block is often closely related to several input words, incorporating the visual block representation can potentially make the prediction of its related words more accurately. Inspired by these observations, we propose a multimodal interaction (MMI) module to learn an image-aware word representation and a word-aware visual representation for each word.

#### 2.3.1 Image-Aware Word Representation

**Cross-Modal Transformer (CMT) Layer:** As shown on the left of Fig. 2.b, to learn better word representations with the guidance of associated images, we first employ an \(m\)-head cross-modal attention mechanism Tsai et al. (2019), by treating \({\bf V}\in\mathbb{R}^{d\times 49}\) as queries, and \({\bf R}\in\mathbb{R}^{d\times(n+1)}\) as keys and values:

\[\textsc{CA}_{i}({\bf V},{\bf R})=\text{softmax}(\frac{[{\bf W}_{ \mathbf{q}}{\bf V}]^{\top}[{\bf W}_{\mathbf{k}}{\bf R}]}{\sqrt{d/m}})[{\bf W}_{ \mathbf{v}_{i}}{\bf R}]^{\top};\] \[\textsc{MH-CA}({\bf V},{\bf R})={\bf W}^{\prime}[\textsc{CA}_{1}({ \bf V},{\bf R}),\ldots,\textsc{CA}_{m}({\bf V},{\bf R})]^{\top},\]

where CA\({}_{i}\) refers to the \(i\)-th head of cross-modal attention, \(\{{\bf W}_{\mathbf{q}_{l}},{\bf W}_{\mathbf{k}_{l}},{\bf W}_{\mathbf{v}_{l}}\} \in\mathbb{R}^{d/m\times d}\), and \({\bf W}^{\prime}\in\mathbb{R}^{d\times d}\) denote the weight matrices for the query, key, value, and multi-head attention, respectively. Next, we stack another three sub-layers on top:

\[\widetilde{{\bf P}}=\text{LN}({\bf V}+\text{MH-CA}({\bf V},{\bf R })); \tag{1}\] \[{\bf P}=\text{LN}(\widetilde{{\bf P}}+\text{FFN}(\widetilde{{\bf P }})), \tag{2}\]

where FFN is the feed-forward network Vaswani et al. (2017), LN is the layer normalization Ba et al. (2016), and \({\bf P}=({\bf p}_{1},{\bf p}_{2},\ldots,{\bf p}_{49})\) is the output representations of the CMT layer.

**Coupled CMT Layer:** However, since the visual representations are treated as queries in the above CMT layer, each generated vector \({\bf p}_{i}\) is corresponding to the \(i\)-th visual block instead of the \(i\)-th input word. Ideally, the image-aware word representation should be corresponding to each word.

To address this, we propose to couple \({\bf P}\) with another CMT layer, which treats the textual representations \({\bf R}\) as queries, and \({\bf P}\) as keys and values. As shown in the top left of Fig. 2.a, this coupled CMT layer generates the final image-aware word representations, denoted by \({\bf A}=({\bf a}_{0},{\bf a}_{1},\ldots,{\bf a}_{n+1})\).

#### 2.3.2 Word-Aware Visual Representation

To obtain a visual representation for each word, it is necessary to align each word with its closely related visual blocks, i.e., assigning high/low attention weights to its related/unrelated visual blocks. Hence, as shown in the right part of Fig. 2.b, we use a CMT layer by treating \({\bf R}\) as queries and \({\bf V}\) as keys and values, which can be considered as a symmetric version of the left CMT layer. Finally, it generates the word-aware visual representations, denoted by \({\bf Q}=({\bf q}_{0},{\bf q}_{1},\ldots,{\bf q}_{n+1})\).

**Visual Gate:** As pointed out in some previous studies Zhang et al. (2018); Lu et al. (2018), it is unreasonable to align many function words such as _the_, _of_, and _well_ with any visual block. Therefore, it is important to incorporate a visual gate to dynamically control the contribution of visual features. Following the practice in previous work, we design a visual gate by combining the information from the above word representations \({\bf A}\) and visual representations \({\bf Q}\) as follows:

\[{\bf g}=\sigma({\bf W}_{a}^{\top}{\bf A}+{\bf W}_{q}^{\top}{\bf Q }), \tag{3}\]

where \(\big{\{}{\bf W}_{a}\), \({\bf W}_{q}\big{\}}\in\mathbb{R}^{d\times d}\) are weight matrices, and \(\sigma\) is the element-wise sigmoid function. Based on the gate output, we can obtain the final word-aware visual representations as \({\bf B}={\bf g}\cdot{\bf Q}\).

#### 2.3.3 CRF Layer

To integrate the word and the visual representations, we concatenate \({\bf A}\) and \({\bf B}\) to obtain the final hidden representations \({\bf H}=({\bf h}_{0},{\bf h}_{1},\ldots,{\bf h}_{n+1})\), where \({\bf h}_{i}\in\mathbb{R}^{2d}\). Following Lample et al. (2016), we then feed \({\bf H}\) to a standard CRF layer, which defines the 

probability of the label sequence \({\bf y}\) given the input sentence \(S\) and its associated image \(V\):

\[P({\bf y}|S,V)=\frac{\exp(score({\bf H},{\bf y}))}{\sum_{{\bf y^{ \prime}}}\exp(score({\bf H},{\bf y^{\prime}}))}; \tag{4}\] \[score({\bf H},{\bf y})=\sum_{i=0}^{n}T_{y_{i},y_{i+1}}+\sum_{i=1}^ {n}E_{{\bf h}_{i},y_{i}};\] (5) \[E_{{\bf h}_{i},y_{i}}={\bf w}_{\text{MNER}}^{y_{i}}\cdot{\bf h}_ {i}, \tag{6}\]

where \(T_{y_{i},y_{i+1}}\) is the transition score from the label \(y_{i}\) to the label \(y_{i+1}\), \(E_{{\bf h}_{i},y_{i}}\) is the emission score of the label \(y_{i}\) for the \(i\)-th word, and \({\bf w}_{\text{MNER}}^{y_{i}}\in\mathbb{R}^{2d}\) is the weight parameter specific to \(y_{i}\).

### Unified Multimodal Transformer

**Motivation:** Since the Multimodal Transformer presented above mainly focuses on modeling the interactions between text and images, it may lead the learnt model to overemphasize the entities highlighted by the image but ignore the remaining entities. To alleviate the bias, we propose to leverage text-based entity span detection (ESD) as an auxiliary task based on the following observation. As ResNet is pre-trained on ImageNet Deng et al. (2009) for the image recognition task, its high-level representations are closely relevant to the final predictions, i.e., the types of contained objects. This indicates that the visual representations from ResNet should be quite useful for identifying types of the detected entities, but are not necessarily relevant to detecting entity spans in the sentence. Therefore, we use purely text-based ESD to guide the final predictions for our main MNER task.

**Auxiliary Entity Span Detection Module:** Formally, we model ESD as another sequence labeling task, and use \({\bf z}\) = \((z_{1},\ldots,z_{n})\) to denote the sequence of labels, where \(z_{i}\in\mathcal{Z}\) and \(\mathcal{Z}=\left\{\mathrm{B},\mathrm{I},\mathrm{O}\right\}\).

As shown in the left part of Fig. 2.a, we employ another Transformer layer to obtain its specific hidden representations as \({\bf T}=({\bf t}_{0},{\bf t}_{1},\ldots,{\bf t}_{n+1})\), followed by feeding it to a CRF layer to predict the probability of the labeled sequence \({\bf z}\) given \(S\):

\[P({\bf z}|S)=\frac{\exp(\sum_{i=0}^{n}T_{zi,z_{i+1}}+\sum_{i=1}^{ n}{\bf w}_{\text{ESP}}^{z}\cdot{\bf t}_{i})}{\sum_{{\bf x^{\prime}}}\exp(\sum_{i=0 }^{n}T_{j^{\prime}_{i},x^{\prime}_{i+1}}+\sum_{i=1}^{n}{\bf w}_{\text{ESP}}^{ z^{\prime}}\cdot{\bf t}_{i})},\]

where \({\bf w}_{\text{ESD}}^{z_{i}}\in\mathbb{R}^{d}\) is the parameter specific to \(z_{i}\).

**Conversion Matrix:** Although ESD is modeled as an auxiliary task separated from MNER, the two tasks are highly correlated since each ESD label should be only corresponding to a subset of labels in MNER. For example, given the sentence in Fig. 2.a, if the first token is predicted to be the beginning of an entity in ESD (i.e., have the label _B_), it should be also the beginning of a typed entity in MNER (e.g., have the label _B-PER_).

To encode such inter-task correspondence, we propose to use a conversion matrix \({\bf W}^{c}\in\mathbb{R}^{|\mathcal{Z}|\times|\mathcal{Y}|}\), where each element \({\bf W}_{j,k}^{c}\) defines the conversion probability from \(\mathcal{Z}_{j}\) to \(\mathcal{Y}_{k}\). Since we have some prior knowledge (e.g., the label \(B\) can only convert to a label subset \(\left\{\textit{B-PER},\textit{B-LOC},\textit{B-ORG},\textit{B-MISC}\right\}\)), we initialize \({\bf W}^{c}\) as follows: if \(\mathcal{Z}_{j}\) is not corresponding to \(\mathcal{Y}_{k}\), \({\bf W}_{j,k}^{c}\) is set to 0; otherwise, \({\bf W}_{j,k}^{c}\) is set to \(\frac{1}{|\mathcal{C}_{j}|},\) where \(\mathcal{C}_{j}\) denotes a subset of \(\mathcal{Y}\) that is corresponding to \(\mathcal{Z}_{j}\).

**Modified CRF Layer for MNER:** After obtaining the conversion matrix, we further propose to fully leverage the text-based entity span predictions to guide the final predictions of MNER. Specifically, we modify the CRF layer for MNER by incorporating the entity span information from ESD into the emission score defined in Eqn. (6):

\[E_{{\bf h}_{i},y_{i}}={\bf w}_{\text{MNER}}^{y_{i}}\cdot{\bf h}_ {i}+{\bf w}_{\text{ESP}}^{z_{i}}\cdot{\bf t}_{i}\cdot{\bf W}_{z_{i},y_{i}}^{ c}. \tag{7}\]

### Model Training

Given a set of manually labeled training samples \(D\) = \(\{S^{j},V^{j},{\bf y}^{j},{\bf z}^{j}\}_{j=1}^{N}\), our overall training objective function is a weighted sum of the sentence-level negative log-likelihood losses for our main MNER task and the auxiliary ESD task2:

Footnote 2: We obtain \({\bf z}^{j}\) by removing the type information in \({\bf y}^{j}\).

\[\mathcal{L}=-\frac{1}{|D|}\sum_{j=1}^{N}\big{(}\log P({\bf y}^{ j}|S^{j},V^{j})+\lambda\log P({\bf z}^{j}|S^{j})\big{)},\]

where \(\lambda\) is a hyperparameter to control the contribution of the auxiliary ESD module.

## 3 Experiments

We conduct experiments on two multimodal NER datasets, comparing our Unified Multimodal Transformer (UMT) with a number of unimodal and multimodal approaches.

\begin{table}
\begin{tabular}{l|r r r|r r r} \hline  & \multicolumn{3}{c|}{TRITTER-2015} & \multicolumn{3}{c}{TWITTER-2017} \\ \cline{2-8}
**Entity Type** & **Train** & **Dev** & **Test** & **Train** & **Dev** & **Test** \\ \hline \hline Person & 2217 & 552 & 1816 & 2943 & 626 & 621 \\ Location & 2091 & 522 & 1697 & 731 & 173 & 178 \\ Organization & 928 & 247 & 839 & 1674 & 375 & 395 \\ Miscellaneous & 940 & 225 & 726 & 701 & 150 & 157 \\ \hline Total & 6176 & 1546 & 5078 & 6049 & 1324 & 1351 \\ \hline \hline
**Num of Teveets** & 4000 & 1000 & 3257 & 3373 & 723 & 723 \\ \hline \end{tabular}
\end{table}
Table 1: The basic statistics of our two Twitter datasets.

 



### Experiment Settings

**Datasets:** We take two publicly available Twitter datasets respectively constructed by Zhang et al. (2018) and Lu et al. (2018) for MNER. Since the two datasets mainly include multimodal user posts published on Twitter during 2014-2015 and 2016-2017, we denote them as TWITTER-2015 and TWITTER-2017 respectively. Table 1 shows the number of entities for each type and the counts of multimodal tweets in the training, development, and test sets of the two datasets3. We have released the two datasets pre-processed by us for research purpose via this link: [https://github.com/jefferyYu/UMT](https://github.com/jefferyYu/UMT).

Footnote 3: The TWITTER-2017 dataset released by Lu et al. (2018) is slightly different from the one used in their experiments, as they later remove a small portion of tweets for privacy issues.

**Hyperparameters:** For each unimodal and multimodal approach compared in the experiments, the maximum length of the sentence input and the batch size are respectively set to 128 and 16. For our UMT approach, most hyperparameter settings follow Devlin et al. (2018) with the following exceptions: (1) the word representations \({\bf C}\) are initialized with the cased _BERTbase_ model pre-trained by Devlin et al. (2018), and fine-tuned during training. (2) we employ a pre-trained 152-layer ResNet4 to initialize the visual representations \({\bf U}\) and keep them fixed during training. (3) For the number of cross-modal attention heads, we set it as \(m\)=12. (4) The learning rate, the dropout rate, and the tradeoff parameter \(\lambda\) are respectively set to 5e-5, 0.1, and 0.5, which can achieve the best performance on the development set of both datasets via a small grid search over the combinations of [1e-5, 1e-4], [0.1, 0.5], and [0.1, 0.9].

Footnote 3: [https://download.pytorch.org/models/resnet152-b12led2d.pth](https://download.pytorch.org/models/resnet152-b12led2d.pth).

### Compared Systems

To demonstrate the effect of our Unified Multimodal Transformer (UMT) model, we first consider a number of representative text-based approaches for NER: (1) _BiLSTM-CRF_(Huang et al., 2015), a pioneering study which eliminates the heavy reliance on hand-crafted features, and simply employs a bidirectional LSTM model followed by a CRF layer for each word's final prediction; (2) _CNN-BiLSTM-CRF_(Ma and Hovy, 2016), a widely adopted neural network model for NER, which is an improvement of _BiLSTM-CRF_ by replacing each word's word embedding with the concatenation of its word embedding and CNN-based character-level word representations; (3) _HBiLSTM-CRF_(Lample et al., 2016), an end-to-end hierarchical LSTM architectures, which replaces the bottom CNN layer in _CNN-BiLSTM-CRF_ with an LSTM layer to obtain the character-level word representations; (4) _BERT_(Devlin et al., 2018), a multi-layer bidirectional Transformer encoder, which gives contextualized representations for each word, followed by stacking a softmax layer for final predictions; (5) _BERT-CRF_, a variant of _BERT_ by replacing the softmax layer with a CRF layer.

Besides, we also consider several competitive multimodal approaches for MNER: (1) _GVATT-HBiLSTM-CRF_(Lu et al., 2018), a state-of-the-art approach for MNER, which integrates _HBiLSTM-CRF_ with the visual context by proposing a visual attention mechanism followed by a visual gate to obtain word-aware visual representations; (2) _AdaCAN-CNN-BiLSTM-CRF_(Zhang et al., 2018), another state-of-the-art approach based on _CNN-BiLSTM-CRF_, which designs an adaptive co-attention network to induce word-aware visual representations for each word; (3) _GVATT-BERT-CRF_

\begin{table}
\begin{tabular}{c|l|r r r r r r r r r r r r r} \hline  & & \multicolumn{10}{c|}{MITITER-2015} & \multicolumn{10}{c|}{Timitters-2017} \\ \cline{3-14} \multirow{3}{*}{**Modality**} & \multirow{3}{*}{**Methods**} & \multicolumn{3}{c|}{**Single Type (f1)**} & \multicolumn{3}{c|}{Overall} & \multicolumn{3}{c|}{Single Type (F1)} & \multicolumn{3}{c}{Overall} \\  & & **REL.** & **LOC.** & **ORG.** & **MISC.** & **P** & **R** & **FI** & **PER.** & **LOC.** & **ORG.** & **MISC.** & **P** & **R** & **Fi** \\ \hline \hline \multirow{4}{*}{Text} & BLSTM-CRF & 26.77 & 72.56 & 41.33 & 28.60 & 38.41 & 61.41 & 60.19 & 64.42 & 85.12 & 72.68 & 72.56 & 52.96 & 74.92 & 74.34 & 76.31 \\  & CNN-BLSTM-CRF & 80.76 & 75.39 & 47.77 & 32.61 & 66.24 & 68.09 & 67.15 & 87.99 & 74.70 & 67.02 & 68.21 & 80.00 & 78.76 & 79.37 \\  & HHILSTM-CRF & 82.34 & 76.83 & 51.59 & 32.52 & 70.32 & 68.05 & 69.17 & 87.91 & 78.57 & 76.67 & 39.32 & 82.69 & 78.16 & 80.37 \\ \cline{1-1} \cline{2-14}  & BERT & 84.72 & 79.91 & 58.36 & 38.81 & 63.30 & 74.61 & 71.32 & 90.88 & 84.00 & 79.25 & 61.63 & 82.19 & 83.72 & 82.95 \\  & BERT-CRF & 84.74 & 80.51 & 60.27 & 37.92 & 69.24 & 75.99 & 71.81 & 90.25 & 83.05 & 81.13 & 62.21 & 83.25 & 85.73 & 83.44 \\ \hline \hline \multirow{4}{*}{TestImage} & GVNT-HBLSTM-CRF & 82.66 & 77.21 & 55.06 & 35.25 & **73.86** & 67.09 & 70.80 & 89.34 & 78.53 & 79.12 & 62.21 & 83.41 & 80.38 & 81.87 \\  & AdaCAN-CNN-BiLSTM-CRF & 81.98 & 78.55 & 53.07 & 34.02 & 72.75 & 68.74 & 70.69 & 89.63 & 77.46 & 92.42 & 67.77 & 84.16 & 80.24 & 82.15 \\ \cline{1-1} \cline{2-14}  & GVAT-BERT-CRF & 84.43 & 80.87 & 59.02 & 38.14 & 69.15 & 74.46 & 71.70 & 90.94 & 83.52 & 81.91 & 62.75 & 83.64 & 84.38 & 84.01 \\ \cline{1-1}  & AAC-CNN-BERT-CRF & 85.28 & 80.64 & 59.39 & 35.88 & 69.87 & 74.59 & 72.15 & 90.20 & 82.97 & **82.67** & 64.83 & 85.13 & 83.20 & 84.10 \\ \cline{1-1}  & **MERT-BERT-CRF (Ours)** & **85.02** & 81.21 & 61.10 & 37.97 & 0.048 & 74.80 & 72.88 & 91.47 & 82.05 & 81.54 & 65.80 & 84.60 & 84.16 & 84.42 \\ \cline{1-1}  & **UMET-BERT-CRF (Ours)** & 85.24 & **81.58** & **63.03** & **39.45** & 71.



and _AdaCAN-BERT-CRF_, our two variants of the above two multimodal approaches, which replace the sentence encoder with _BERT_; (4) _MT-BERT-CRF_, our Multimodal Transformer model introduced in Section 2.3; (5) _UMT-BERT-CRF_, our unified architecture by incorporating the auxiliary entity span detection module into Multimodal Transformer, as introduced in Section 2.4.

All the neural models are implemented with PyTorch, and all the experiments are conducted on _NVIDIA RTX 2080 Ti_ GPUs.

### Main Results

In Table 2, we report the precision (**P**), recall (**R**), and F1 score (**F1**) achieved by each compared method on our two Twitter datasets.

First, comparing all the text-based approaches, we can clearly observe that _BERT_ outperforms the other compared methods with a significant margin on both datasets. Moreover, it is easy to see that empowering BERT with a CRF layer can further boost the performance. All these observations indicate that the contextualized word representations are indeed quite helpful for the NER task on social media texts, due to the context-aware characteristics. This agrees with our first motivation.

Second, comparing the state-of-the-art multimodal approaches with their corresponding unimodal baselines, we can find that the multimodal approaches can generally achieve better performance, which demonstrates that incorporating the visual context is generally useful for NER. Besides, we can see that although _GVATT-HBiLSTM-CRF_ and _AdaCAN-CNN-BiLSTM-CRF_ can significantly outperform their unimodal baselines, the performance gains become relatively limited when replacing their sentence encoder with _BERT_. This suggests the challenge and the necessity of proposing a more effective multimodal approach.

Third, in comparison with the two existing multimodal methods, our Multimodal Transformer _MT-BERT-CRF_ outperforms the state-of-the-art by 2.5% and 2.8% respectively, and also achieves better performance than their BERT variants. We conjecture that the performance gains mainly come from the following reason: the two multimodal methods only focus on obtaining word-aware visual representations, whereas our _MT-BERT-CRF_ approach targets at generating both image-aware word representations and word-aware visual representations for each word. These observations are in line with our second motivation.

Finally, comparing all the unimodal and multimodal approaches, it is clear to observe that our Unified Multimodal Transformer (i.e., _UMT-BERT-CRF_) can achieve the best performance on both datasets, outperforming the second best methods by 1.14% and 1.05%, respectively. This demonstrates the usefulness of the auxiliary entity span detection module, and indicates that the auxiliary module can help our Multimodal Transformer alleviate the bias brought by the associated images, which agrees with our third motivation.

### Ablation Study

To investigate the effectiveness of each component in our Unified Multimodal Transformer (UMT) architecture, we perform comparison between the full UMT model and its ablations with respect to the auxiliary entity span detection (ESD) module and the multimodal interaction (MMI) module.

As shown in Table 3, we can see that all the components in UMT make important contributions to the final results. On the one hand, removing the whole ESD module will significantly drop the performance, which shows the importance of alleviating the visual bias. In particular, discarding the conversion matrix in the ESD module also leads to the performance drop, which indicates the usefulness of capturing the label correspondence between the auxiliary module and our main MNER task.

On the other hand, as the main contribution of 

our MMI module, Image-Aware Word Representations (WR) demonstrates its indispensable role in the final performance due to the moderate performance drop after removal. Besides, removing the visual gate also results in minor performance drop, indicating its importance to the full model.

### Further Analysis

**Importance of MMI and ESD Modules:** To better appreciate the importance of two main contributions (i.e., MMI and ESD modules) in our proposed approaches, we conduct additional analysis on our two test sets. In Fig. 3 and Fig. 4, we show the number of entities that are wrongly/correctly predicted by BERT-CRF, but correctly/wrongly predicted by each multimodal method5.

Footnote 5: Note that here we use strict matches (i.e., correct span and type predictions).

First, we can see from Fig. 3 that with the MMI module, our _MT-BERT-CRF_ and _UMT-BERT-CRF_ approaches correctly identify more entities, compared with the two multimodal baselines. Table 4.A shows a specific example. We can see that our two methods correctly classify the type of _Wolf Hall_ as _MISC_ whereas the compared systems wrongly predict its type as _LOC_, probably because our MMI module enforces the image-aware word representations of _Wolf Hall_ to be closer to drama names.

Second, in Fig. 4, it is clear to observe that compared with the other three methods, _UMT-BERT-CRF_ can significantly decrease the bias brought by the visual context due to incorporating our auxiliary ESD module. In Table 4.B, we show a concrete example: since _Game of Thrones_ is ignored by the image, the two multimodal baselines fail to identify them; in contrast, with the help of the auxiliary ESD module, _UMT-BERT-CRF_ successfully eliminates the bias.

**Effect of Incorporating Images:** To obtain a better understanding of the general effect of incorporating associated images into our MNER task, we carefully examine our test sets and choose two representative test samples to compare the prediction results of different approaches.

First, we observe that most improvements gained by multimodal methods come from those samples where the textual contents are informal or incomplete but the visual context provides useful clues. For example, in Table 4.C, we can see that without the visual context, _BERT-CRF_ fails to identify that the two entities refer to two singers in the concert, but all the multimodal approaches can correctly classify their types after incorporating the image.

Second, by manually checking the test set of our two datasets, we find that in around 5% of the social media posts, the associated images might be irrelevant to the textual contents due to two kinds of reasons: (1) these posts contain image memes, cartoons, or photos with metaphor; (2) their images and textual contents reflect different aspects of the same event. In such cases, we observe that multimodal approaches generally perform worse than _BERT-CRF_. A specific example is given in Table 4.D, where all the multimodal methods wrongly classify _Siri_ as _PER_ because of the unrelated face in the image.

## 4 Related Work

As a crucial component of many information extraction tasks including entity linking [9], opinion mining [10], and event detection [67], named

\begin{table}
\begin{tabular}{l l l l l} \hline \hline Importance of the MMI Module & Importance of the ESD Module & Importance of Associated Images & Noise of Associated Images \\ \hline \hline BERT-CRF: & 1-ICOC, 2-ICOC & 1-PEAR, 2-IMSC✓, 3-ORG✓ & 1-MASC, 2-ORG✓ & 1-MASC✓ \\ AdaCAN-BERT-CRF: 1-LOCP, 2-ICOC & 1-PEAR, 2-NONE, 3-ORG✓ & 1-PERR✓, 2-PRR✓ & 1-PERR✓ \\ MT-BERT-CRF: & 1-MISC✓, 2-MICS✓ & 1-PEAR✓, 2-NONE✓, 3-ORG✓ & 1-PEAR✓, 2-PRR✓ & 1-PERR✓ \\ UMT-BERT-CRF: & 1-MISC✓, 2-MICS✓ & 1-PERR✓, 2-MSC✓, 3-ORG✓ & 1-PERR✓, 2-PRR✓ & 1-PERR✓ \\ \hline \hline \end{tabular}
\end{table}
Table 4: The second row shows several representative samples together with their manually labeled entities in the test set of our two TWITTER datasets, and the bottom four rows show predicted entities of different methods on these test samples.

 

entity recognition (NER) has attracted much attention in the research community in the past two decades Li et al. (2018).

**Methods for NER:** In the literature, various supervised learning approaches have been proposed for NER. Traditional approaches typically focus on designing various effective NER features, followed by feeding them to different linear classifiers such as maximum entropy, conditional random fields (CRFs), and support vector machines Chieu and Ng (2002); Florian et al. (2003); Finkel et al. (2005); Ratinov and Roth (2009); Lin and Wu (2009); Passos et al. (2014); Luo et al. (2015). To reduce the feature engineering efforts, a number of recent studies proposed to couple different neural network architectures with a CRF layer Lafferty et al. (2001) for word-level predictions, including convolutional neural networks Collobert et al. (2011), recurrent neural networks Chiu and Nichols (2016); Lample et al. (2016), and their hierarchical combinations Ma and Hovy (2016). These neural approaches have been shown to achieve the state-of-the-art performance on different benchmark datasets based on formal text Yang et al. (2018).

However, when applying these approaches to social media text, most of them fail to achieve satisfactory results. To address this issue, many studies proposed to exploit external resources (e.g., shallow parser, Freebase dictionary, and orthographic characteristics) to incorporate a set of tweet-specific features into both traditional approaches Ritter et al. (2011); Li et al. (2014); Baldwin et al. (2015) and recent neural approaches Limsopatham and Collier (2016); Lin et al. (2017), which can obtain much better performance on social media text.

**Methods for Multimodal NER (MNER):** As multimodal data become increasingly popular on social media platforms, several recent studies focus on the MNER task, where the goal is to leverage the associate images to better identify the named entities contained in the text. Specifically, Moon et al. (2018) proposed a multimodal NER network with modality attention to fuse the textual and visual information. To model the inter-modal interactions and filter out the noise in the visual context, Zhang et al. (2018) and Lu et al. (2018) respectively proposed an adaptive co-attention network and a gated visual attention mechanism for MNER. In this work, we follow this line of work. But different from them, we aim to propose an effective multi-modal method based on the recent Transformer architecture Vaswani et al. (2017). To the best of our knowledge, this is the first work to apply Transformer to the task of MNER.

## 5 Conclusion

In this paper, we first presented a Multimodal Transformer architecture for the task of MNER, which captures the inter-modal interactions with a multimodal interaction module. Moreover, to alleviate the bias of the visual context, we further proposed a Unified Multimodal Transformer (UMT), which incorporates an entity span detection module to guide the final predictions for MNER. Experimental results show that our UMT approach can consistently achieve the best performance on two benchmark datasets.

There are several future directions for this work. On the one hand, despite bringing performance improvements over existing MNER methods, our UMT approach still fails to perform well on social media posts with unmatched text and images, as analyzed in Section 3.5. Therefore, our next step is to enhance UMT so as to dynamically filter out the potential noise from images. On the other hand, since the size of existing MNER datasets is relatively small, we plan to leverage the large amount of unlabeled social media posts in different platforms, and propose an effective framework to combine them with the small amount of annotated data to obtain a more robust MNER model.

#### Acknowledgments

We would like to thank three anonymous reviewers for their valuable comments. This research is supported by the National Research Foundation, Singapore under its International Research Centres in Singapore Funding Initiative, and the Natural Science Foundation of China under Grant 61672288. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of National Research Foundation, Singapore.

## References

* Ba et al. (2016) Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. _arXiv preprint arXiv:1607.06450_ 

Timothy Baldwin, Marie-Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. In _Proceedings of the Workshop on Noisy User-generated Text_, pages 126-135.
* Chieu and Ng (2002) Hai Leong Chieu and Hwee Tou Ng. 2002. Named entity recognition: a maximum entropy approach using global information. In _Proceedings of COLING_.
* Chiu and Nichols (2016) Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional lstm-cnns. _Transactions of the Association for Computational Linguistics_, 4:357-370.
* Collobert et al. (2011) Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _Journal of Machine Learning Research_, 12:2461-2505.
* Deng et al. (2009) Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In _2009 IEEE conference on computer vision and pattern recognition_, pages 248-255.
* Derczynski et al. (2015) Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marike Van Erp, Genevieve Gorrell, Raphael Truncy, Johann Petrak, and Kalina Bontcheva. 2015. Analysis of named entity recognition and linking for tweets. _Information Processing & Management_, 51(2):32-49.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Finkel et al. (2005) Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In _Proceedings of ACL_.
* Florian et al. (2003) Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. Named entity recognition through classifier combination. In _Proceedings of NAACL_.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of CVPR_, pages 770-778.
* Huang et al. (2015) Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. _arXiv preprint arXiv:1508.01991_.
* Lafferty et al. (2001) John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of ICML_.
* Lample et al. (2016) Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In _Proceedings of NAACL-HLT_.
* Li et al. (2014) Chenliang Li, Aixin Sun, Jianshu Weng, and Qi He. 2014. Tweet segmentation and its application to named entity recognition. _IEEE Transactions on knowledge and data engineering_, 27(2):558-570.
* Li et al. (2012) Chenliang Li, Jianshu Weng, Qi He, Yuxia Yao, Anwitaman Datta, Aixin Sun, and Bu-Sung Lee. 2012. Twiner: named entity recognition in targeted twitter stream. In _Proceedings of SIGIR_.
* Li et al. (2018) Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2018. A survey on deep learning for named entity recognition. _arXiv preprint arXiv:1812.09449_.
* Limsopatham and Collier (2016) Nut Limsopatham and Nigel Collier. 2016. Bidirectional LSTM for named entity recognition in twitter messages. In _Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT)_.
* Lin et al. (2017) Bill Yuchen Lin, Frank F Xu, Zhiyi Luo, and Konny Zhu. 2017. Multi-channel bilstm-crf model for emerging named entity recognition in social media. In _Proceedings of the 3rd Workshop on Noisy User-generated Text_.
* Lin and Wu (2009) Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In _Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP_.
* Lu et al. (2018) Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In _Proceedings of ACL_, pages 1990-1999.
* Luo et al. (2015) Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Zaiqing Nie. 2015. Joint entity recognition and disambiguation. In _Proceedings of EMNLP_.
* Ma and Hovy (2016) Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In _Proceedings of ACL_.
* Maynard et al. (2012) Diana Maynard, Kalina Bontcheva, and Dominic Rout. 2012. Challenges in developing opinion mining tools for social media. In _Proceedings of the@ NLP can u tag usergeneratedcontent_.
* Moon et al. (2018) Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity recognition for short social media posts. In _Proceedings of NAACL_.
* Passos et al. (2014) Alexandre Passos, Vineet Kumar, and Andrew McLulum. 2014. Lexicon infused phrase embeddings for named entity resolution. In _Proceedings of CoNLL_.
* Ratinov and Roth (2009) Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In _Proceedings of CoNLL_ 

Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: an experimental study. In _Proceedings of ACL_.
* Ritter et al. (2012) Alan Ritter, Oren Etzioni, and Sam Clark. 2012. Open domain event extraction from twitter. In _Proceedings of SIGKDD_.
* Sang and Veenstra (1999) Erik F Sang and Jorn Veenstra. 1999. Representing text chunks. In _Proceedings of eACL_.
* Torisawa et al. (2007) Kentaro Torisawa et al. 2007. Exploiting wikipedia as external knowledge for named entity recognition. In _Proceedings of EMNLP-CoNLL_.
* Tsai et al. (2019) Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In _Proceedings of ACL_.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Proceedings of NIPS_, pages 5998-6008.
* Yadav and Bethard (2018) Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In _Proceedings of COLING_.
* Yang et al. (2018) Jie Yang, Shuailong Liang, and Yue Zhang. 2018. Design challenges and misconceptions in neural sequence labeling. In _Proceedings of COLING_.
* Zhang et al. (2018) Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In _Proceedings of AAAI_, pages 5674-5681.
* Zhou and Su (2002) GuoDong Zhou and Jian Su. 2002. Named entity recognition using an hmm-based chunk tagger. In _Proceedings of ACL_.