Venue: NAACL 2022

# ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition

Xinyu Wang\({}^{\diamond 1}\), Min Gui\({}^{\heartsuit}\), Yong Jiang\({}^{\spadesuit}\)+, Zixia Jia\({}^{\diamond}\), Nguyen Bach\({}^{\clubsuit}\), Tao Wang, Zhongqiang Huang\({}^{\clubsuit}\), Fei Huang\({}^{\clubsuit}\), Kewei Tu\({}^{\diamond*}\)

\({}^{\diamond}\)School of Information Science and Technology, ShanghaiTech University

\({}^{\diamond}\)Shanghai Engineering Research Center of Intelligent Vision and Imaging

\({}^{\clubsuit}\)DAMO Academy, Alibaba Group

\({}^{\heartsuit}\)Shopee, Singapore

\({}^{\clubsuit}\)Microsoft

{wangxy1,jiazx,tukw}@shanghaitech.edu.cn, min.gui@shopee.com

{yongjiang.yy,z.huang,f.huang}@alibaba-inc.com

nguyenbach@microsoft.com

###### Abstract

Recently, Multi-modal Named Entity Recognition (MNER) has attracted a lot of attention. Most of the work utilizes image information through region-level visual representations obtained from a pretrained object detector and relies on an attention mechanism to model the interactions between image and text representations. However, it is difficult to model such interactions as image and text representations are trained separately on the data of their respective modality and are not aligned in the same space. As text representations take the most important role in MNER, in this paper, we propose **I**mage-**t**ext **A**lignments (ITA) to align image features into the textual space, so that the attention mechanism in transformer-based pretrained textual embeddings can be better utilized. ITA first aligns the image into regional object tags, image-level captions and optical characters as visual contexts, concatenates them with the input texts as a new cross-modal input, and then feeds it into a pretrained textual embedding model. This makes it easier for the attention module of a pretrained textual embedding model to model the interaction between the two modalities since they are both represented in the textual space. ITA further aligns the output distributions predicted from the cross-modal input and textual input views so that the MNER model can be more practical in dealing with text-only inputs and robust to noises from images. In our experiments, we show that ITA models can achieve state-of-the-art accuracy on multi-modal Named Entity Recognition datasets, even without image information.1

Footnote 1: This work was done when Xinyu Wang, Min Gui and Nguyen Bach were at Alibaba Group.

2022/19/18/2022/2022/2022/19/17

## 1 Introduction

Named Entity Recognition (NER) [Sundheim1995] has attracted increasing attention in natural language processing community. It has been applied to a lot of domains such as news [Tjong Kim Sang2002, Tjong Kim Sang and De Meulder2003], E-commerce [Fetahu et al.2021], social media [Strauss et al.2016, Derczynski et al.2017] and bio-medicine [Dogan et al.2014, Li et al.2016]. Several recent studies focus on improving the accuracy of NER models through utilizing image information (MNER) in tweets [Zhang et al.2018, Moon et al.2018, Lu et al.2018]. Most approaches to MNER use the attention mechanism to model the interaction between image and text representations [Yu et al.2020, Zhang et al.2021a, Sun et al.2021], in which image representations are from a pretrained feature extractor, i.e. ResNet [He et al.2016], and text representations are extracted from pretrained textual embeddings, i.e. BERT [Devlin et al.2019]. Since these models are separately trained on datasets of different modalities and their feature representations are not aligned, it is difficult for the attention mechanism to model the interaction between the two modalities.

Recently, pretrained vision-language (V+L) models such as LXBERT [Tan and Bansal2019], UNITER [Chen et al.2020] and Oscar [Li et al.2020b] have achieved significant improvement on several cross-modal tasks such as image captioning, VQA [Agrawal et al.2015], NLVR [Young et al.2014] and image-text retrieval [Suhr et al.2019]. Most pretrained V+L models are trained on image-text pairs and simply concatenate text features and image features as the input of pretraining. There are, however, two problems. First, texts in these datasets mainly contain common nouns

instead of named entities2 which leads to an inductive bias over common nouns and images. Second, despite its important role in pretraining V+L models, the image modality only plays an auxiliary role in MNER for disambiguation, and can sometimes even be discarded. These problems make pretrained V+L models perform weaker than pretrained language models for MNER.

Footnote 2: [https://visualgenome.org/data_analysis/statistics](https://visualgenome.org/data_analysis/statistics)

Pretrained textual embeddings such as BERT, XLM-RoBERTa Conneau et al. (2020) and LUKE Yamada et al. (2020) have achieved state-of-the-art performance on various NER datasets through simple fine-tuning of pretrained textual embeddings. Since most of the transformer-based pretrained textual embeddings are trained over long texts, recent work Akbik et al. (2019); Schweter and Akbik (2020); Yamada et al. (2020); Wang et al. (2021) has shown that introducing document-level contexts can significantly improve the accuracy of a NER model. The attention mechanism in transformer-based pretrained textual embeddings can utilize contexts to improve the token representation of a sequence. Moreover, pretrained V+L models such as Oscar and VinVL Zhang et al. (2021) can use object tags detected in images to significantly ease the alignments between text and image features. Therefore, the images in MNER can be converted to texts as well so that the image representations can be aligned to the space of text representations. As a result, the attention module of the pretrained textual embeddings have the capability to easily model the interactions between aligned image and text representations, without introducing a new attention module. In this paper, we propose ITA, a simple but effective framework for **I**mage-**T**ext **A**lignments. ITA converts an image into visual contexts in textual space by multi-level alignments. We concatenate the NER texts with the visual contexts as a new cross-modal input view and then feed it into a pretrained textual embedding model to improve the token representations of NER texts, which are fed into a linear-chain CRF Lafferty et al. (2001) layer for prediction. In practice, a MNER model should be robust when there is only text information, as images may be unavailable or can introduce noises. Sometimes it is even undesirable to use images as image feature extraction can be inefficient in online serving. Therefore, we further propose to utilize the cross-modal input view to improve the accuracy of textual input view, based on cross-view alignment that minimizes the KL divergence over the probability distributions of the two views.

ITA can be summarized in four aspects:

1. Object Tags as Local Alignment: ITA locally extracts object tags and its corresponding attributes of image regions from an object detector.
2. Image Captions as Global Alignment: ITA summarizes what the image is describing through predicting image captions from an image captioning model.
3. Optical Character Alignment: ITA extracts the texts presented in the image via optical character recognition (OCR).
4. Cross-View Alignment: we calculate the KL divergence between the output distributions of two input views.

We show in experiments that ITA can significantly improve the model accuracy on MNER datasets and achieve the state-of-the-art. The cross-view alignment module can significantly improve both the cross-modal and textual input views, and bridge the performance gap between the two views.

## 2 Approaches

We consider the NER task as a sequence labeling problem. Given a sentence \({\mathbf{w}}=\{w_{1},\cdots,w_{n}\}\) with \(n\) tokens and its corresponding image \(I\), an sequence labeling model aims to predict a label sequence \({\mathbf{y}}=\{y_{1},\cdots,y_{n}\}\) at each position. In our framework, we focus on incorporating visual information to improve the representations of the input tokens by aligning visual and textual information effectively. We use a visual context generator to convert the image \(I\) into texts forming visual contexts \({\mathbf{w}^{\prime}}=\{w^{\prime}_{1},\cdots,w^{\prime}_{m}\}\) with \(m\) tokens. We then concatenate the input text and visual contexts as a cross-modal text+image (**I+T**) input view instead of the text (**T**) input view. We feed the **I+T** input into a pretrained textual embeddings model to get stronger token representations of the input sentence. Then the token representations are fed into a linear-chain CRF layer to get the label sequence \({\mathbf{y}}\). To further improve the model accuracy of both input views, we use the cross-view alignment module to align the output distributions of **I+T** and 

input views during training. The architecture of our framework is shown in Figure 1.

### NER Model Architecture

We use a neural model with a linear-chain CRF layer, a widely used approach for the sequence labeling problem Huang et al. (2015); Akbik et al. (2018); Devlin et al. (2019). The input is fed into a transformer-based pretrained textual embeddings model and the output token representations \(\{\mathbf{r}_{1},\cdots,\mathbf{r}_{n}\}\) are fed into the CRF layer:

\[p_{\theta}(\mathbf{y}|\mathbf{w})=\frac{\prod\limits^{n}_{i=1}\psi(y_{ i-1},y_{i},\mathbf{r}_{i})}{\sum\limits_{\mathbf{y}^{\prime}\in\mathcal{Y}(\mathbf{w})} \prod\limits^{n}_{i=1}\psi(y^{\prime}_{i-1},y^{\prime}_{i},\mathbf{r}_{i})}\]

where \(\theta\) is the model parameters, \(\mathcal{Y}(\mathbf{w})\) is the set of all possible label sequences given the input \(\mathbf{w}\). Given the gold label sequence \(\hat{\mathbf{y}}\) in the training data, the objective function of the model for the **T** input view is:

\[\mathcal{L}_{\text{T}}(\theta)=-\log p_{\theta}(\hat{\mathbf{y}}|\mathbf{ w}) \tag{1}\]

The loss can be calculated using Forward algorithm.

### Image-text Alignments

The transformer-based pretrained textual embeddings have strong representations over texts. Therefore, ITA converts the image information into textual space through generating texts from the image so that the learning of the self-attention in the transformer-based model can be significantly eased compared with simply using image features from an object detector. We propose a local (LA), a global (GA) and an optical character alignment (OCA) approaches for alignments.

Object Tags as Local AlignmentGiven an image, the image information can be decomposed into a set of objects in local regions. The object tags of each region textually describe the local information in the image. To extract the objects, we use an object detector **OD** to identify and locate the objects in the image:

\[\mathbf{a},\mathbf{o}=\textbf{OD}(I);\text{where}\] \[\mathbf{a}=\{\mathbf{a}_{1},\mathbf{a}_{2},\cdots,\mathbf{a}_{l}\}~{}\text{and}~{ }\mathbf{o}=\{o_{1},o_{2},\cdots,o_{l}\}\]

The attribute predictions from the object detector contain multiple attribute tags \(\mathbf{a}_{i}\) for each object \(o_{i}\). We linearize and sort the objects in a descending order based on the confidences of the detection model. For each object, we heuristically keep 0 to 3 attributes with confidence scores above a threshold \(m\). We linearize the attributes and put the attributes before the corresponding objects since the attributes are the adjectives describing the object tags. As a result, we take the predicted \(l\) object tags \(\mathbf{o}\) and their attribute tags \(\mathbf{a}\) from the object detector as the locally aligned visual contexts \(\mathbf{w}^{\text{LA}}\):

\[\mathbf{w}^{\text{LA}}=\{\mathbf{a}_{1},o_{1},\mathbf{a}_{2},o_{2},\cdots,\mathbf{a}_{l},o_{ l}\}\]

Image Captions as Global AlignmentThough the local alignment can localize the image into

Figure 1: The architecture of ITA. ITA aligns an image into object tags, image captions and texts from OCR. ITA takes them as visual contexts and then feeds them together with the input texts into the transformer-based embeddings. In the cross-view alignment module, ITA minimizes the distance between the output distribution of cross-modal inputs and textual inputs.

 

objects, the objects cannot fully describe the of the whole image. Image captioning is a task that predicts the meaning of an image. Therefore, we align the image into \(k\) image captions by an image captioning model **IC**:

\[\{\mathbf{w}^{1},\mathbf{w}^{2},\cdots,\mathbf{w}^{k}\}=\textbf{IC}(I)\]

where \(\{\mathbf{w}^{1},\mathbf{w}^{2},\cdots,\mathbf{w}^{k}\}\) are captions generated from beam search with \(k\) beams. We concatenate the \(k\) captions together with a special separate token [X] to form the aligned global visual contexts \(\mathbf{w}^{\text{GA}}\):

\[\mathbf{w}^{\text{GA}}{=}[\mathbf{w}^{1},\texttt{[X]},\mathbf{w}^{2},\texttt{[X]},\cdots, \texttt{[X]},\mathbf{w}^{k}]\]

The exact label (e.g. "[SEP]" in BERT) of the special [X] token depends on the selection of embeddings.

Optical Character AlignmentSome image contain text when they are created to enrich the semantic information that the images want to convey. In order to better understand this type of image, we use an **OCR** model to identify and extract the texts in the image:

\[\mathbf{w}^{\text{OCA}}=\textbf{OCR}(I)\]

where \(\mathbf{w}^{\text{OCA}}\) are the texts extracted by the **OCR** model. Note that \(\mathbf{w}^{\text{OCA}}\) may be an empty text if there is no text in the image.

We concatenate the input sentence and our aligned visual contexts to form the **I+T** input view \(\hat{\mathbf{w}}=[\mathbf{w};\mathbf{w}^{\prime}]\), where \(\mathbf{w}^{\prime}\) can be one of \(\mathbf{w}^{\text{LA}}\), \(\mathbf{w}^{\text{GA}}\), \(\mathbf{w}^{\text{OCA}}\) or the concatenation of all (we denote it as **AII**). The transformer-based embeddings are fed with the **I+T** input view and then output image-text fused token representations for each token \(\{\mathbf{r}_{1}^{\prime},\cdots,\mathbf{r}_{n}^{\prime}\}\). The token representations are fed into the CRF layer to get the probability distribution \(p_{\theta}(\mathbf{y}|\hat{\mathbf{w}})\). Similar to Eq. 1, the objective function of the model for the **I+T** input view is:

\[\mathcal{L}_{\text{I+T}}(\theta)=-\log p_{\theta}(\hat{\mathbf{y}}|\hat{\mathbf{w}}) \tag{2}\]

Cross-View AlignmentThere are several limitations in incorporating images into NER prediction: 1) the images may not available in testing; 2) aligning images to texts requires several pipelines in pre-processing instead of an end-to-end manner, which is so time-consuming that it is not applicable to some time-critical scenes such as online serving; 3) the noises in the image can mislead the MNER model to make wrong predictions. To alleviate these issues, we propose Cross-View Alignment (CVA), which targets at reducing the gap between the **I+T** and **T** input views over the output distributions so that the MNER model can better utilize the textual information in the input. During training, CVA minimizes the KL divergence over the probability distribution of **I+T** and **T** input views:

\[\mathcal{L}_{\text{CVA}}(\theta){=}\text{KL}(p_{\theta}(\mathbf{y}|\hat{\mathbf{w}})|| p_{\theta}(\mathbf{y}|\mathbf{w})) \tag{3}\]

Since the **I+T** input view has additional visual information in the input and we want the **T** input view to match the accuracy of **I+T** input view, we only back-propagate through \(p_{\theta}(\mathbf{y}|\mathbf{w})\) in Eq. 3. Therefore, Eq. 3 is equivalent to calculating the cross-entropy loss over the two distributions:

\[\mathcal{L}_{\text{CVA}}(\theta){=}\sum_{\mathbf{y}\in\mathcal{Y}(\mathbf{x})}p_{ \theta}(\mathbf{y}|\hat{\mathbf{w}})\log p_{\theta}(\mathbf{y}|\mathbf{w}) \tag{4}\]

As the set of all possible label sequences \(\mathcal{Y}(\mathbf{x})\) is exponential in size, we calculate the posterior distributions of each position \(p_{\theta}(y_{i}|\mathbf{w})\) and \(p_{\theta}(y_{i}|\hat{\mathbf{w}})\) through forward-backward algorithm to approximate Eq. 4:

\[p_{\theta}(y_{k}|*){\propto}\sum_{\{y_{0},\dots,y_{k-1}\}}\prod_{ i=1}^{k}\psi(y_{i-1},y_{i},\mathbf{r}_{i}^{*})\] \[{\times}\sum_{\{y_{k+1},\dots,y_{n}\}}\prod_{i=k+1}^{n}\psi(y_{i- 1},y_{i},\mathbf{r}_{i}^{*})\] \[\mathcal{L}_{\text{CVA}}(\theta){=}\sum_{i=1}^{n}p_{\theta}(y_{i }|\hat{\mathbf{w}})\log p_{\theta}(y_{i}|\mathbf{w})) \tag{5}\]

where \(\mathbf{r}_{i}^{*}\) represents either \(\mathbf{r}_{i}\) or \(\mathbf{r}_{i}^{\prime}\).

TrainingDuring training, we jointly train **T** and **I+T** input views with the training objective in Eq. 1 and 2 together with the CVA alignment training objective in Eq. 5. As a result, the final training objective for ITA is:

\[\mathcal{L}_{\text{ITA}}=\mathcal{L}_{\text{CVA}}+\mathcal{L}_{\text{T}}+ \mathcal{L}_{\text{I+T}}\]

## 3 Experiments

We conduct experiments on two MNER datasets. To show the effectiveness of our approaches, we use two embedding settings and compare our approaches with previous multi-modal approaches 



### Settings

DatasetsWe show the effectiveness of our approaches on Twitter-15, Twitter-17 and SNAP Twitter datasets3 containing 4,000/1,000/3,357, 3,373/723/723 and 4,290/1,432/1,459 sentences in train/development/test split respectively. The Twitter-15 dataset is constructed by Zhang et al. (2018). The SNAP dataset is constructed by Lu et al. (2018) and the Twitter-17 dataset is a filtered version of SNAP constructed by Yu et al. (2020).

Footnote 3: Twitter-15 and 17 datasets are available at [https://github.com/jefferyYu/UMT](https://github.com/jefferyYu/UMT).

Model ConfigurationFor token representations, we use BERT base model to fairly compare with most of the recent work (Yu et al., 2020; Zhang et al., 2021a; Sun et al., 2021). Recently, XLM-RoBERTa has achieved state-of-the-art accuracy on various NER datasets by feeding the input together with contexts to the model. To further utilize the visual contexts in transformer-based embeddings, we use XLM-RoBERTa large (XLMR) model as another embedding in our experiments. To extract object tags and image captions of the image, we use VinVL (Zhang et al., 2021b), which is a pre-trained V+L model based on a newly pretrained large-scale object detector based on the ResNeXt-152 C4 architecture. We use the object detection module of VinVL to predict object tags and their corresponding attributes. The number of object tags and attributes varies over the images and is no more than \(100\). We set the threshold \(m\) to be \(0.1\) for keeping the attributes of each object. For image captions, we use VinVL large model finetuned on MS-COCO (Lin et al., 2014) captions4 with CIDEr optimization (Rennie et al., 2017). In our experiments, we use a beam size of \(5\) with at most 20 tokens for prediction and keep all the \(5\) captions as the visual contexts. For OCR, we use Tesseract OCR5Smith (2007), which is an open source OCR engine. We use the default configuration of the engine to extract texts in the image6.

Footnote 4: github.com/cincrosoft/Oscar5github.com/tesseract-ocr/tesseract

Footnote 6: Please refer to Appendix A.2 for more statistics.

Training ConfigurationDuring training, we finetune the pretrained textual embedding model by AdamW (Loshchilov and Hutter, 2018) optimizer. In experiments we use the grid search to find the learning rate for the embeddings within \([1\times 10^{-6},5\times 10^{-4}]\). For BERT embeddings, we finetune the embeddings with a learning rate of \(5\times 10^{-5}\) with a batch size of \(16\). For XLMR embeddings, we use a learning rate of \(5\times 10^{-6}\) and a batch size of \(4\) instead. For the learning rate of the CRF layer, we use a grid search over \([0.05,0.5]\) and \([0.005,0.05]\) for BERT and XLMR respectively. The MNER models are trained for \(10\) epochs and we report the average results from \(5\) runs with different random seeds for each setting.

### Results

In Table 1, we compare our approaches with our baselines with different training and evaluation modalities (**T** for the text-only input view and **I+T**

\begin{table}
\begin{tabular}{l|l||c c|c c} \hline \multicolumn{1}{c||}{} & & \multicolumn{1}{c|}{Twitter-15} & \multicolumn{1}{c}{Twitter-17} & SNAP \\ \cline{3-6} \multirow{2}{*}{Moral} & \multirow{2}{*}{Approach} & **Eval** & **Eval** & **Eval** \\  & & **Modal** & **Modal** & **Modal** \\  & & **T****i+T** & **T****i+T** & **T****i+T** & **T****i+T** \\ \hline \hline \multicolumn{6}{c}{**BERT-CRF**} \\ \hline T & **BERT-CRF** & 74.79 & - & 85.18 & - & 85.98 & - \\ \hline  & **TA-LA** & – & 75.18 & - & 85.67 & - & 86.26 \\  & **ITA-GA** & – & 75.17 & - & 85.75 & - & 86.72 \\  & **ITA-OCAL** & – & 75.01 & - & 85.64 & - & 86.52 \\  +**TT** & **TA-All** & – & 75.15 & - & 85.78 & - & 86.79 \\  & **ITA-LA\_CCA** & – & 75.26 & 75.20 & 85.72 & 85.62 & 86.51 & 86.41 \\  & **ITA-CA\_CCA** & – & 75.45 & 75.52 & 85.96 & **85.85** & 86.42 & 86.39 \\  & **ITA-OCAL\_CCA** & – & 75.26 & 75.30 & 85.73 & 85.79 & 86.64 & 86.59 \\  & **ITA-ALL\_ccx** & **75.67 & **75.60** & **85.98** & 85.72 & **86.83** & **86.75** \\ \hline \multicolumn{6}{c}{**CLMR-CRF**} \\ \hline T & **XLMR-CRF** & 77.37 & - & 88.73 & - & 89.39 & - \\ \hline  & **ITA-LA** & – & 77.64 & - & 89.29 & - & 89.68 \\  & **ITA-GA** & – & 77.78 & - & 89.23 & - & 89.78 \\  & **ITA-OCAL** & – & 77.94 & - & 89.31 & - & 89.64 \\ I+T & **ITA-All** & – & 77.81 & - & 89.62 & - & 90.10 \\  & **ITA-LA\_CCA** & 77.87 & 77.93 & 89.45 & **89.90** & 89.85 & 89.91 \\  & **ITA-GA\_CCA** & 78.03 & 78.02 & 89.41 & 89.62 & 89.85 & 90.09 \\  & **ITA-OCAL\_CCA** & 77.57 & 77.59 & 89.32 & 89.55 & 89.90 & 89.84 \\  & **ITA-all\_CCx** & **78.25 & **78.03** & **89.47** & 89.75 & **90.02** & **90.15** \\ \hline \end{tabular}
\end{table}
Table 1: A comparison of ITA and our baseline.

\begin{table}
\begin{tabular}{l||c|c|c} \hline Approach & Twitter-15 & Twitter-17 & SNAP \\ \hline \multicolumn{3}{c}{**Reported F1**} \\ \hline

* & 86.80 \\
* & 87.40 \\
* \\ \hline \multicolumn{3}{c}{**Our Reproductions**} \\ \hline
**BERT-CRF** & 74.79 & 85.18 & 85.98 \\

* & 62.14 \\ Ours: **ITA-All\_CCA** & **76.01** & **86.45** & **87.44** \\ \hline \end{tabular}
\end{table}
Table 2: A comparison of our approaches and state-of-the-art approaches. \(\blacktriangle\): Wu et al. (2020); †: results are from Yu et al. (2020); ‡: Sun et al. (2020), \(\blacklozenge\): Sun et al. (2021), note that **RbBERT‡**‡** uses the test set to select the best model; \(\diamond\): results are from Zhang et al. (2021a).


for the multi-modal input view). Results show that ITA models are significantly stronger than our **BERT-CRF** and **XLMR-CRF** baselines (Student's t-test with \(p<0.05\)). For the aligned visual contexts, LA, GA and OCA are competitive in most of the cases. To show the effectiveness of CVA, we report the evaluation results of both input views in evaluation. With CVA, the accuracy of both input views can be improved, especially the **T** input view. CVA can improve the **T** input view to be competitive with **I+T** input view. Moreover, the combination of all the alignments **ITA-All\({}_{\text{tCVA}}\)** can further improve the model accuracy in most of the cases. The accuracy of the MNER models can be significantly improved if we use XLMR embeddings, which shows the importance of the text modality in MNER. With XLMR embeddings, the model accuracy can be further improved with ITA. The relative improvements over the baseline models are sometimes higher with XLMR than with BERT, which shows that the visual contexts can be further utilized with stronger embeddings.

In Table 2, we compare **ITA** with previous state-of-the-art approaches. For previous approaches, we report the results including **OCSGA**, **UMT**, **RIVA**, **RpBERT**, **UMGF**, which are the proposed approaches of Wu et al. (2020); Yu et al. (2020); Sun et al. (2021); Wang et al. (2021) and Zhang et al. (2021a) respectively. For fair comparison, we report the results of these models based on the BERT base embeddings. Moreover, since most of these previous approaches report the best model accuracy instead of the averaged model accuracy, we use the best model accuracy of **ITA-All\({}_{\text{tCVA}}\)** over \(5\) runs. We also report our reproduced results of **UMT**, **RpBERT** and **UMGF** on the corresponding datasets. The results show that **ITA-All\({}_{\text{tCVA}}\)** outperforms all of the previous approaches. On the SNAP dataset, the reported accuracy of **RpBERT\({}_{\textbf{base}}\)** is competitive with **ITA-All\({}_{\text{tCVA}}\)**. However, we find that the accuracy of our reproduced **RpBERT\({}_{\textbf{base}}\)7** is significantly lower than the reported accuracy, even after careful check of the source code and hyper-parameter tuning. Moreover, the fact that our **BERT-CRF** baseline achieves competitive accuracy with previous state-of-the-art multi-modal approaches shows that most of the previous work has not fully explored the strength of the text representations for the task.

Footnote 7: We reproduced the results based on the official code for **RpBERT\({}_{\textbf{base}}\)**: [https://github.com/Multimodal-NER/RpBERT](https://github.com/Multimodal-NER/RpBERT)

Discussion about Textual ModulesAs we have shown in Table 1 and 2, the textual baselines (i.e. **BERT-CRF**) of previous work are significantly lower than that of ours. In most of the previous MNER architectures, the textual modules are mainly based on the baseline architectures with some modifications. We further show the baselines of previous work are not well-trained and how the multi-modal approaches perform with stronger textual modules. In Table 3, we rerun the **BERT-CRF** baseline based on the released codes of **UMT8**. Based on the code of **UMT**, we tried to improve the baseline models in the code by using the same loss function as ours9. The accuracy of **BERT-CRF** models in the code are significantly improved but the **UMT** models based on the improved code are not improved and even get worse in Twitter-17. Therefore, we suspect the **UMT** model cannot be further improved even with stronger textual modules. Zhang et al. (2021a) also reported the baseline based on the implementation of Yu et al. (2020), so we suspect the **UMGF** model cannot be improved as well. Therefore, the under-trained textual baselines of previous work make the effectiveness of the images unclear and we show that some of the MNER models perform even weaker than our **BERT-CRF** model.

Footnote 8: [https://github.com/jefferyYu/UMT](https://github.com/jefferyYu/UMT)

Footnote 9: The details are discussed in Appendix A.5

### Comparison with Other Variants

To further show the effectiveness of ITA, we perform several comparisons between ITA and the following variants of the MNER model in Table 4:

ITA-Random:We generate random image-text pairs for the model. For each sentence, we randomly select the image in the dataset and generate the corresponding visual contexts. The noises of random visual contexts make the model accuracy

\begin{table}
\begin{tabular}{l||c|c} \hline Approaches & Twitter-15 & Twitter-17 \\ \hline \hline
**BERT-CRF\({}_{\text{UTM}}\)** & 71.81 & 83.44 \\
**BERT-CRF\({}_{\textbf{Ours}}\)** & 74.79 & 85.18 \\ \hline  & \multicolumn{2}{c}{**Our Reproductions**} \\ \hline
**BERT-CRF\({}_{\text{UTM}}\)** & 71.74 & 84.20 \\
**BERT-CRF\({}_{\text{UMT-improved}}\)** & 72.53 & 84.48 \\
**UMT** & 72.83 & 84.88 \\
**UMT\({}_{\text{Improved}}\)** & 72.96 & 84.50 \\ \hline \end{tabular}
\end{table}
Table 3: Our reproductions of previous baselines and approaches. “Improved” means our improved models based on the UMT code base.

 

drop slightly comparing with our **BERT-CRF** baseline, which shows the improvement of our approach is from the visual contexts rather than extending the input sequence length the embeddings.

ITA-Joint:It is an ablated model of \({\bf ITA}\)-\({\bf All_{+CVA}}\). We train the \({\bf IIA}\)-\({\bf All}\) model for both input views without the CVA loss in Eq. 5. The model accuracy is improved moderately with only the \({\bf T}\) input view while our \({\bf ITA}\)-\({\bf All_{+CVA}}\) can improve both input views significantly, which shows the effectiveness of the CVA module of ITA.

\({\bf ITA}\)-\({\bf LA}_{\bf BU}\) and \({\bf IIA}\)-\({\bf GA}_{\bf BU}\):We conduct experiments to see how the accuracy changes when using weaker image features. We use Bottom-Up features proposed by Anderson et al. (2018) for object detection and image captioning. The captioning model is a pretrained image captioning model10 proposed by Luo et al. (2018) with the Bottom-Up features and self-critical training (Rennie et al., 2017). Results show that there is no significant difference between the visual contexts from Bottom-Up features and VinVL features. Therefore, our approaches can utilize other off-the-shelf vision models to extract visual contexts.

Footnote 10: [https://github.com/ruotianluo/self-critical](https://github.com/ruotianluo/self-critical) ̇pytorch

Ita-OCApaddle:We conduct experiments to see how the accuracy changes when using stronger OCR models. We use PaddleOCR11 for the experiment, which is one of the newest open resource lightweight OCR system. Results show that the model accuracy can be slightly improved comparing with \({\bf ITA}\)-OCA, which shows the ITA models can be improved by using better OCR models.

Footnote 11: [https://github.com/PaddlePaddle/PaddleOCR](https://github.com/PaddlePaddle/PaddleOCR)

Bert-CRF*ImgFeat:Instead of \({\bf IT}\)\({\bf A}\), we can directly feed the image region features generated from an object detector into the BERT. We use ResNet-152 model to generate region features and then feed the features into a linear layer to project the region features into the same space of text features in the BERT. Moreover, we compare the model with **RbBERT w/o Rp**, which is an ablated model of **RbBERT** and is equivalent to **BERT-CRF+imgFeat** over the usage of BERT embeddings. Sun et al. (2021) showed **RPBERT w/o Rp** can improve the model accuracy compared with their baseline. However, our results show that the model accuracy slightly drops comparing with our **BERT-CRF**, which shows that it is difficult for the attention module of BERT to learn the relations of the unaligned representations of two modalities.

VinVL-CRF:To show how the pretrained V+L models perform on the NER task, we use VinVL since it is a very recent state-of-the-art pretrained V+L model on a lot of multi-modal tasks. We feed the VinVL model with texts and images in the MNER datasets and finetune the model over the task. We take the text representations output from VinVL as the input of the CRF layer. The accuracy of the finetuned VinVL model drops significantly compared to the BERT model, which shows that the inductive bias of the pretrained V+L model hurts the model accuracy on MNER.

Bert+ViNL-CRF:As the VinVL model may lead to an inductive bias over the common nouns and the image, we jointly finetune the BERT and VinVL models and concatenate the output text representations of the two models. The accuracy is improved on a moderate scale, which shows BERT is complementary to VinVL for MNER.

\begin{table}
\begin{tabular}{l||c c|c c} \hline  & Twitter-15 & Twitter-17 & SNAP \\ \hline \multirow{3}{*}{Approach} & **Eval** & **Eval** & **Eval** \\  & **Modal** & **Modal** & **Modal** \\  & **T** & **1+T** & **T** & **1+T** & **1+T** \\ \hline \end{tabular} 
\begin{tabular}{l||c c|c c} \hline \multicolumn{4}{l}{} \\ \hline \hline \({\bf ITA}\)-Random & - & 74.67 & - & 84.98 & - & 85.82 \\ \({\bf ITA}\)-GABU & - & 75.10 & - & 85.77 & - & 86.51 \\ \({\bf ITA}\)-LA\({}_{\bf AU}\) & - & 75.18 & - & 85.59 & - \\ \({\bf ITA}\)-OCARedite & - & 75.12 & - & 85.87 & - & 86.66 \\ \({\bf BERT}\)-CRF\({}_{\textbf{simPerat}}\) & - & 74.70 & - & 84.99 & - & 85.90 \\ \({\bf VinVL}\)-CRF & - & 60.58 & - & 75.55 & - & 74.53 \\ \({\bf BERT}\)+VinVL-CRF & - & 74.89 & - & 85.19 & - & 86.14 \\ \({\bf ITA}\)-Joint & 74.88 & 75.22 & 85.31 & 85.60 & 86.06 & 86.34 \\ \hline \multicolumn{4}{l}{**References**} \\ \hline \multicolumn{4}{l}{**RepBERT w/o Rp**} & - & 72.60 & - & - & - & 86.20 \\ \({\bf ITA}\)-All\({}_{\text{CVA}}\) & **75.50** & **75.41** & **85.89** & **85.84** & **86.83** & **86.75** \\ \hline \end{tabular}
\end{table}
Table 4: A comparison of other variants of MNER models.

Figure 2: A relation between the number of captions input to the MNER model and model accuracy. The x-axis is the number of captions. The y-axis is the averaged F1 score on the test set.

 



### Analysis

Effect of the Number of CaptionsUsing more captions output from the captioning model can improve diversities of the visual contexts but can add noises to them as well. To better understand how the number of captions affects the model accuracy, we change the beam size and keep all the sentences output from the captioning model. The trends in Figure 2 show that the model accuracy increases until \(5\) captions for all the datasets and gradually drops when the number of captions further increases for Twitter-15 and 17 datasets. The observation shows that using \(5\) captions keeps a good balance between the diversities and correctness of the captions.

How ITA Eases the Cross-Modal AlignmentsPrevious work such as Moon et al. (2018); Sun et al. (2021) visualized modality attention in several cases to show the effectiveness of their approaches. However, visualizing the multi-layer attention in transformer-based embeddings is relatively difficult. Instead of studying special cases, we statistically calculate the averaged L2 distance between token representations \({\mathbf{r}_{i}}\) and \({\mathbf{r}^{\prime}_{i}}\) from two input modalities to show how the token representations depend on image information. In Figure 3, the L2 distance **ITA-All** is significantly larger than that of **BERT-CRF+ImgFeat**. Besides, the standard deviation of **BERT-CRF+ImgFeat** is very large. The observations show the image region features make the alignment become difficult and unstable while our visual contexts can significantly ease the cross-modal alignments. Moreover, with CVA, the L2 distance becomes much smaller and stable as CVA aligns the two input views to reduce the dependence on images, which shows the MNER model can better utilize the textual information with CVA.

How Images Affect the NER PredictionTo study the effectiveness of the images over each label, we show a comparison between our model and our baselines in Table 5. When the relative improvement of the F1 score is larger than \(0.5\), the relative improvement of precision is larger than that of recall. The observation shows that the main improvement of MNER is mainly because the images can help the model to reduce false-positive predictions for disambiguation on uncertain entities.12

Footnote 12: In Appendix A.3, we show several cases to show the effectiveness of ITA to affect NER prediction.

## 4 Related Work

Multi-modal Named Entity RecognitionMost of the previous approaches to MNER focus on the interaction between image and text features through attention mechanisms. Moon et al. (2018) proposed a modality attention network to fuse the text and image features before the input to the BiLSTM layer. Lu et al. (2018) additionally used a visual attention gate for the output features of the BiLSTM layer. Zhang et al. (2018) proposed an adaptive co-attention network after the BiLSTM layer to model the interaction between image and text. Recently, Wu et al. (2020) proposed OCSGA, which use object labels to model the interaction between image and object labels in an additional dense co-attention layer. Compared with the work, we show a simpler and more effective way to utilize object labels and additionally use other alignment approaches to further improve the model accuracy. Yu et al. (2020) proposed UMT, which utilized a multi-modal interaction module and an auxiliary entity span detection module for MNER. Zhang et al. (2021a) proposed UMGF, which utilizes a pretrained parser to create the graph connection between visual object tags and textual words. They used a graph attention network to fuse the textual and visual features. In order to better model whether the image is related to the text, Sun et al. (2021) proposed RpBERT, which additionally trains on a text-image relation classification dataset proposed by Vempala and Preotiuc-Pietro (2019) to prevent the negative effect of noisy images. Comparing with RpBERT, we use CVA to let the NER model better utilize the input sentences without such kinds of supervision. All of these approaches focus on fusing the image and text features through the attention mechanism but ignore the gap between the image and text features while we propose to fully utilize the attention mechanism in the pretrained textual embeddings through

Figure 3: Averaged L2 distance between the token representations without image input (\({\mathbf{r}_{i}}\)) and with image input (\({\mathbf{r}^{\prime}_{i}}\)). The error bars mean the standard deviation over \(5\) runs.



aligning image features into textual space. Besides, some cross-media research also shows the effectiveness of OCR texts [321, 45] and object tags [60] have been shown. Most of the approaches introduced a new attention module over cross-modal features while in comparison ITA effectively utilizes the attention module in the pretrained textual embeddings.

Pretrained Vision-Language ModelsInspired by related work on language model pretraining, visual-language pretraining (VLP) has recently attracted a lot of attention [9, 8, 7, 10, 11, 12, 13]. The pretrained V+L models are pretrained on large-scale image-text pairs and have achieved state-of-the-art accuracy over various vision-language tasks such as image captioning, VQA, NLVR and image-text retrieval. Recently, Li et al. (2020a) proposed Oscar to add object tags in pretraining so that self-attention can learn the image-text alignments easily. Following Oscar, Zhang et al. (2021b) proposed VinVL to train a large-scale object detector to improve the pretrained V+L model's accuracy. Comparing with VLP, MNER is a totally different task. Firstly, the image-caption pairs are given in VLP and the image and text are equally important in pretraining for general representations. Therefore, using global alignment is meaningless for VLP but makes sense for MNER. In MNER, the input text is not the caption of the image and the image may not adds additional information to the input text. Secondly, though captions and object tags are often utilized in VLP, how to effectively utilize the captions and object tags of the image in MNER is rarely considered. Finally, besides the local and global alignments, another aspect of ITA is the optical character alignment and cross-view alignment, which is rarely considered in VLP.

## 5 Conclusion

In this paper, we propose Image-Text Alignments for multi-modal named entity recognition, which convert images into object labels, captions and OCR texts to align the image representations into textual space in a multi-level manner and form a cross-modal input view. The model can effectively utilize attention module of the transformer-based embeddings. Considering noises, availability of images and inference speed for practical use, we propose cross-view alignment, which let the MNER models better utilize the text information in the input. In our experiments, we show that ITA significantly outperforms previous state-of-the-art approaches on MNER datasets. We also show that most of the previous work failed to train a good textual baseline while our textual baseline can easily match or even outperform previous multi-modal approaches. In analysis, we further analyze how ITA eases the cross-modal alignments and how the images affect the NER prediction.

## Acknowledgements

This work was supported by the National Natural Science Foundation of China (61976139) and by Alibaba Group through Alibaba Innovative Research Program.

## References

* [Agrawal et al.2018] Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. L. Zitnick, Devi Parikh, and Dhruv 

Batra. 2015. Vqa: Visual question answering. International Journal of Computer Vision, 123:4-31.
* Akbik et al. (2019) Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 724-728, Minneapolis, Minnesota. Association for Computational Linguistics.
* Akbik et al. (2018) Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Anderson et al. (2018) Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In _CVPR_.
* Chen et al. (2016) Tao Chen, Xiangnan He, and Min-Yen Kan. 2016. Context-aware image tweet modelling and recommendation. In _Proceedings of the 24th ACM international conference on Multimedia_, pages 1018-1027.
* Chen et al. (2020) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In _European conference on computer vision_.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Derczynski et al. (2017) Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In _Proceedings of the 3rd Workshop on Noisy User-generated Text_, pages 140-147, Copenhagen, Denmark. Association for Computational Linguistics.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dolgan et al. (2014) Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. _Journal of biomedical informatics_, 47:1-10.
* Fetahu et al. (2021) Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021. Gazetteet enhanced named entity recognition for code-mixed web queries. In _SIGIR '21_, SIGIR '21, New York, NY, USA. Association for Computing Machinery.
* He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 770-778.
* Huang et al. (2015) Zhiheng Huang, W. Xu, and Kailiang Yu. 2015. Bidirectional lstm-crf models for sequence tagging. _ArXiv_, abs/1508.01991.
* Lafferty et al. (2001) John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In _Proceedings of the Eighteenth International Conference on Machine Learning_, ICML-'01, page 282-289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
* Li et al. (2016) Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. _Database: The Journal of Biological Databases and Curation_, 2016.
* Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. _arXiv preprint arXiv:1908.03557_.
* Li et al. (2020a) Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furru Wei, Yejin Choi, and Jianfeng Gao. 2020a. Oscar: Object-semantic s aligned pre-training for vision-language tasks. In _ECCV_.
* Li et al. (2020b) Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b. Oscar: Object-semantics aligned pre-training for vision-language tasks. In _European Conference on Computer Vision_, pages 121-137. Springer.
* Lin et al. (2014) Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In _European conference on computer vision_, pages 740-755. Springer.
* Loshchilov and Hutter (2018) Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In _International Conference on Learning Representations_.
* Lu et al. (2018) Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, 

pages 1990-1999, Melbourne, Australia. Association for Computational Linguistics.
* Lu et al. (2019) Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In _NeurIPS_.
* Luo et al. (2018) R. Luo, Brian L. Price, Scott D. Cohen, and Gregory Shakhnarovich. 2018. Discriminability objective for training descriptive captions. _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 6964-6974.
* Moon et al. (2018) Seungwan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity recognition for short social media posts. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 852-860, New Orleans, Louisiana. Association for Computational Linguistics.
* Ronnie et al. (2017) Steven J Ronnie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In _Proceedings of the IEEE conference on computer vision and pattern recognition_, pages 7008-7024.
* Schweter and Akbik (2020) Stefan Schweter and Alan Akbik. 2020. Flert: Document-level features for named entity recognition. _arXiv preprint arXiv:2011.06993_.
* Smith (2007) Ray Smith. 2007. An overview of the tesseract ocr engine. In _Ninth international conference on document analysis and recognition (ICDAR 2007)_, volume 2, pages 629-633. IEEE.
* Strauss et al. (2016) Benjamin Strauss, Bethany Toma, Alan Ritter, Marie-Catherine de Marneffe, and Wei Xu. 2016. Results of the WNUT16 named entity recognition shared task. In _Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT)_, pages 138-144, Osaka, Japan. The COLING 2016 Organizing Committee.
* Suhr et al. (2019) Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In _ACL_.
* Sun et al. (2020) Lin Sun, Jiquan Wang, Yindu Su, Fangsheng Weng, Yuxuan Sun, Zengwei Zheng, and Yuanyi Chen. 2020. RIVA: A pre-trained tweet multimodal model based on text-image relation for multimodal NER. In _Proceedings of the 28th International Conference on Computational Linguistics_, pages 1852-1862, Barcelona, Spain (Online). International Committee on Computational Linguistics.
* Sun et al. (2021) Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fangsheng Weng. 2021. RPBERT: A text-image relation propagation-based bert model for multimodal ner. In _AAAI_.
* Sundheim (1995) Beth M. Sundheim. 1995. Named entity task definition, version 2.1. In _Proceedings of the Sixth Message Understanding Conference_, pages 319-332.
* Tan and Bansal (2019) Hao Hao Tan and M. Bansal. 2019. LXmert: Learning cross-modality encoder representations from transformers. In _EMNLP_.
* Tsjong Kim Sang (2002) Erik F. Tsjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In _COLING-02: The 6th Conference on Natural Language Learning 2002 (CoNLL-2002)_.
* Yang et al. (2003) Erik F. Tsjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In _Conceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_, pages 142-147.
* Vempala and Preojuc-Pietro (2019) Alakananda Vempala and Daniel Preojuc-Pietro. 2019. Categorizing and inferring the relationship between the text and image of Twitter posts. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2830-2840, Florence, Italy. Association for Computational Linguistics.
* Wang et al. (2021) Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Improving named entity recognition by external context retrieving and cooperative learning. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1800-1812, Online. Association for Computational Linguistics.
* Wang et al. (2020) Yue Wang, Jing Li, Michael Lyu, and Irwin King. 2020. Cross-media keyphrase prediction: A unified framework with multi-modality multi-head attention and image wordings. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 3311-3324, Online. Association for Computational Linguistics.
* Wu et al. (2016) Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, and Anton Van Den Hengel. 2016. What value do explicit high level concepts have in vision to language problems? In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 203-212.
* Wu et al. (2020) Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-fung Leung, and Qing Li. 2020. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts. ACM MM.
* Yamada et al. (2020) Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity-aware self-attention. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 6442-6454, Online. Association for Computational Linguistics.
* Young et al. (2014) Peter Young, Alice Lai, M. Hodosh, and J. Hockennaier. 2014. From image descriptions to visual denotations: 

New similarity metrics for semantic inference over event descriptions. _Transactions of the Association for Computational Linguistics_, 2:67-78.
* Yu et al. (2021) Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. In _AAAI_.
* Yu et al. (2020) Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 3342-3352, Online. Association for Computational Linguistics.
* Zhang et al. (2021a) Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021a. Multimodal graph fusion for named entity recognition with targeted visual guidance. In _Proceedings of the AAAI Conference on Artificial Intelligence_.
* Zhang et al. (2021b) Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021b. Vinvl: Revisiting visual representations in vision-language models. In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_, pages 5579-5588.
* Zhang et al. (2018) Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In _Thirty-Second AAAI Conference on Artificial Intelligence_.

## Appendix A Appendix

### Details of Experiment Settings

We run our code on Tesla V100 GPU with 16 GB memory. It takes about two hours to train a model. The size of model parameter is approximately equal to size of BERT/XLMR embeddings.

### Details of OCA

Table 6 shows that the OCR system only finds about 26% sentences have texts in the image and the extracted texts have an average of 28 tokens. The statistics show that ITA-OCA can help to improve the model accuracy with only 26% of the samples have OCR texts.

### Case Study

Despite that images can generally help to improve the accuracy of the NER model, there are a lot of cases that the images may contain misleading information to hurt the model prediction. We study two cases for LA nad GA: 1) the entities are wrongly predicted by **BERT-CRF** baseline but are correctly predicted by **ITA**; 2) the entities are wrongly predicted by **ITA** without CVA but are correctly predicted by the baseline and **ITA** with CVA. Figure 4 shows the two cases with two samples for each. Figure 4 (a) shows the first case, which shows the importance of the visual contexts. The baseline model failed to recognize the person entities "TWICE" and "Harry Potter" possibly because the two words are usually an adverp and a book name respectively. For the **I+T** input view, our MNER model is able to recognize the hints such as "two girls", "young girl", "a couple of young men" and "woman" in the visual contexts and then correctly predict the two entities. Figure 4 (b) shows the second case, which shows how the noises from the image mislead the model predictions. There are three- and two-person entities in gold labels but the visual contexts indicate that the top right image has "two baseball players" and the bottom right image has only "a woman". As a result, **ITA** without CVA only predict two and one person entities according to the visual contexts in the two samples respectively. However, with CVA, **ITA** takes a good balance in utilizing the textual and visual information and correctly predicts the entity labels in both **T** and **I+T** input views.

For OCA, we study how the extracted texts can help model prediction. In the upper sample of Figure 5, there are two "Donald" words in the image. The baseline model failed to identify the latter one while **ITA-OCA** can successfully identify both of them. In the bottom of Figure 5, the texts in the image are mainly talking about "HARRY STYLES", which helps the model prediction.

### Discussion

In our paper, we use the captioning and object detection model based on MSCOCO and visual genome. The model performance could be improved if we use domain-specific models (Twitter domain). For OCA, the model accuracy may be poor if the OCR system does not support a certain language.

### Loss Function Comparison with UMT

In the codes of UMT, the BERT embeddings tokenize the token in a sentence into subtokens. The codes use the first subtoken as the token representation to predict the corresponding label. However, for the other subtokens, the codes use a special label "PAD" for prediction. Therefore, the target labels are changed. For example, the original label 

* [164] M. C. Bader, J. S. Hubert, and A. K. Grafner, "Theoretical evaluation of the performance of a new machine learning approach to the problem of interest in the search engine," _Journal of Machine Learning Research_**2015**(3):1-7, 2018.
* [165] D. R. Bishop, "A simple algorithm for solving problems with applications to machine learning," _Machine Learning_**2019**(1):1-10, 2019.
* [166] P. L. Barnett, "On the complexity of the solution of the problem of interest," _IEEE Transactions on Information Theory and Applications_**2016**(1):1-10, 2016.
* [167] T. B. Cover, "An introduction to boosting algorithms," _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_, pages 1-10, 2002.

[MISSING_PAGE_POST]

 

sequence is "B-X, I-X, O, B-X, O, O" but now it becomes "B-X, PAD, PAD, I-X, O, B-X, O, PAD, O". As a result, the exact training objective changes compared with the training objective in the paper of UMT. We improve the code by removing all the "PAD" labels and just use the first subtoken of each token as the token representation. Our improved baseline model is significantly improved, while the accuracy of UMT model in the improved code cannot be further improved.

\begin{table}
\begin{tabular}{l||c|c|c}  & Twitter-15 & Twitter-17 & SNAP \\ \hline Num Sents w/ OCR / Total Sents & 2,049 / 8,288 (24.72\%) & 1,197 / 4,461 (26.83\%) & 1,869 / 7,181 (26.03\%) \\ Avg. Length & 27.72 & 27.00 & 28.93 \\ \hline \end{tabular}
\end{table}
Table 6: A statistic about the number of sentences has OCR texts and the average length of OCR texts.