SlowGuess/ACL_2020_OCR huggingface dataset:
Modality	Methods	TWITTER-2015	TWITTER-2017
PER.	Single Type (F1)	P	Overall	PER.	Single Type (F1)	P	Overall
LOC.	ORG.	MISC.	LOC.	ORG.	MISC.
Text	BiLSTM-CRF	76.77	72.56	41.33	26.80	68.14	61.09	64.42	85.12	72.68	72.50	52.56	79.42	73.43	76.31
CNN-BiLSTM-CRF	80.86	75.39	47.77	32.61	66.24	68.09	67.15	87.99	77.44	74.02	60.82	80.00	78.76	79.37
HBiLSTM-CRF	82.34	76.83	51.59	32.52	70.32	68.05	69.17	87.91	78.57	76.67	59.32	82.69	78.16	80.37
BERT	84.72	79.91	58.26	38.81	68.30	74.61	71.32	90.88	84.00	79.25	61.63	82.19	83.72	82.95
BERT-CRF	84.74	80.51	60.27	37.29	69.22	74.59	71.81	90.25	83.05	81.13	62.21	83.32	83.57	83.44
Text+Image	GVATT-HBiLSTM-CRF	82.66	77.21	55.06	35.25	73.96	67.90	70.80	89.34	78.53	79.12	62.21	83.41	80.38	81.87
AdaCAN-CNN-BiLSTM-CRF	81.98	78.95	53.07	34.02	72.75	68.74	70.69	89.63	77.46	79.24	62.77	84.16	80.24	82.15
GVATT-BERT-CRF	84.43	80.87	59.02	38.14	69.15	74.46	71.70	90.94	83.52	81.91	62.75	83.64	84.38	84.01
AdaCAN-BERT-CRF	85.28	80.64	59.39	38.88	69.87	74.59	72.15	90.20	82.97	82.67	64.83	85.13	83.20	84.10
MT-BERT-CRF (Ours)	85.30	81.21	61.10	37.97	70.48	74.80	72.58	91.47	82.05	81.84	65.80	84.60	84.16	84.42
UMT-BERT-CRF (Ours)	85.24	81.58†	63.03†	39.45†	71.67	75.23	73.41†	91.56†	84.73†	82.24	70.10†	85.28	85.34	85.31†
Table 2: Performance comparison on our two TWITTER datasets. † indicates that UMT-BERT-CRF is significantly better than GVATT-BERT-CRF and AdaCAN-BERT-CRF with p-value < 0.05 based on paired t-test.
3.1 Experiment Settings
Datasets: We take two publicly available Twitter datasets respectively constructed by Zhang et al. (2018) and Lu et al. (2018) for MNER. Since the two datasets mainly include multimodal user posts published on Twitter during 2014-2015 and 2016-2017, we denote them as TWITTER-2015 and TWITTER-2017 respectively. Table 1 shows the number of entities for each type and the counts of multimodal tweets in the training, development, and test sets of the two datasets<sup>3</sup>. We have released the two datasets preprocessed by us for research purpose via this link: https://github.com/jefferyYu/UMT.
Hyperparameters: For each unimodal and multimodal approach compared in the experiments, the maximum length of the sentence input and the batch size are respectively set to 128 and 16. For our UMT approach, most hyperparameter settings follow Devlin et al. (2018) with the following exceptions: (1) the word representations  $\mathbf{C}$  are initialized with the cased  $BERT_{base}$  model pre-trained by Devlin et al. (2018), and fine-tuned during training. (2) we employ a pre-trained 152-layer ResNet4 to initialize the visual representations  $\mathbf{U}$  and keep them fixed during training. (3) For the number of cross-modal attention heads, we set it as  $m = 12$ . (4) The learning rate, the dropout rate, and the tradeoff parameter  $\lambda$  are respectively set to 5e-5, 0.1, and 0.5, which can achieve the best performance on the development set of both datasets via a small grid search over the combinations of [1e-5, 1e-4], [0.1, 0.5], and [0.1, 0.9].
3.2 Compared Systems
To demonstrate the effect of our Unified Multimodal Transformer (UMT) model, we first consider a number of representative text-based approaches for NER: (1) BiLSTM-CRF (Huang et al., 2015), a pioneering study which eliminates the heavy reliance on hand-crafted features, and simply employs a bidirectional LSTM model followed by a CRF layer for each word's final prediction; (2) CNN-BiLSTM-CRF (Ma and Hovy, 2016), a widely adopted neural network model for NER, which is an improvement of BiLSTM-CRF by replacing each word's word embedding with the concatenation of its word embedding and CNN-based character-level word representations; (3) HBiLSTM-CRF (Lample et al., 2016), an end-to-end hierarchical LSTM architectures, which replaces the bottom CNN layer in CNN-BiLSTM-CRF with an LSTM layer to obtain the character-level word representations; (4) BERT (Devlin et al., 2018), a multi-layer bidirectional Transformer encoder, which gives contextualized representations for each word, followed by stacking a softmax layer for final predictions; (5) BERT-CRF, a variant of BERT by replacing the softmax layer with a CRF layer.
Besides, we also consider several competitive multimodal approaches for MNER: (1) GVATT-HBiLSTM-CRF (Lu et al., 2018), a state-of-the-art approach for MNER, which integrates HBiLSTM-CRF with the visual context by proposing a visual attention mechanism followed by a visual gate to obtain word-aware visual representations; (2) AdaCAN-CNN-BiLSTM-CRF (Zhang et al., 2018), another state-of-the-art approach based on CNN-BiLSTM-CRF, which designs an adaptive coattention network to induce word-aware visual representations for each word; (3) GVATT-BERT-CRF
3The TWITTER-2017 dataset released by Lu et al. (2018) is slightly different from the one used in their experiments, as they later remove a small portion of tweets for privacy issues. https://download.pytorch.org/models/resnet152-b121ed2d.pth.
Methods	TWITTER-2015	TWITTER-2017
P	R	F1	P	R	F1
UMT-BERT-CRF	71.67	75.23	73.41	85.28	85.34	85.31
w/o ESD Module	70.48	74.80	72.58	84.60	84.16	84.42
w/o Conversion Matrix	70.43	74.98	72.63	84.72	84.97	84.85
w/o Image-Aware WR	70.33	75.44	72.79	83.83	85.94	84.87
w/o Visual Gate	71.34	75.15	73.19	85.31	84.68	84.99
Table 3: Ablation Study of Unified Multimodal Transformer.
and AdaCAN-BERT-CRF, our two variants of the above two multimodal approaches, which replace the sentence encoder with BERT; (4) MT-BERT-CRF, our Multimodal Transformer model introduced in Section 2.3; (5) UMT-BERT-CRF, our unified architecture by incorporating the auxiliary entity span detection module into Multimodal Transformer, as introduced in Section 2.4.
All the neural models are implemented with PyTorch, and all the experiments are conducted on NVIDIA RTX 2080 Ti GPUs.
3.3 Main Results
In Table 2, we report the precision (P), recall (R), and F1 score (F1) achieved by each compared method on our two Twitter datasets.
First, comparing all the text-based approaches, we can clearly observe that BERT outperforms the other compared methods with a significant margin on both datasets. Moreover, it is easy to see that empowering BERT with a CRF layer can further boost the performance. All these observations indicate that the contextualized word representations are indeed quite helpful for the NER task on social media texts, due to the context-aware characteristics. This agrees with our first motivation.
Second, comparing the state-of-the-art multimodal approaches with their corresponding unimodal baselines, we can find that the multimodal approaches can generally achieve better performance, which demonstrates that incorporating the visual context is generally useful for NER. Besides, we can see that although GVATT-HBiLSTM-CRF and AdaCAN-CNN-BiLSTM-CRF can significantly outperform their unimodal baselines, the performance gains become relatively limited when replacing their sentence encoder with BERT. This suggests the challenge and the necessity of proposing a more effective multimodal approach.
Third, in comparison with the two existing multimodal methods, our Multimodal Transformer MT-BERT-CRF outperforms the state-of-the-art by  $2.5\%$  and  $2.8\%$  respectively, and also achieves bet
Figure 3: The number of entities (shown in y-axis) that are incorrectly predicted by BERT-CRF, but get corrected by each multimodal method
Figure 4: The number of entities (shown in y-axis) that are correctly predicted by BERT-CRF, but wrongly predicted by each multimodal method
ter performance than their BERT variants. We conjecture that the performance gains mainly come from the following reason: the two multimodal methods only focus on obtaining word-aware visual representations, whereas our MT-BERT-CRF approach targets at generating both image-aware word representations and word-aware visual representations for each word. These observations are in line with our second motivation.
Finally, comparing all the unimodal and multimodal approaches, it is clear to observe that our Unified Multimodal Transformer (i.e., UMT-BERT-CRF) can achieve the best performance on both datasets, outperforming the second best methods by  $1.14\%$  and  $1.05\%$ , respectively. This demonstrates the usefulness of the auxiliary entity span detection module, and indicates that the auxiliary module can help our Multimodal Transformer alleviate the bias brought by the associated images, which agrees with our third motivation.
To investigate the effectiveness of each component in our Unified Multimodal Transformer (UMT) architecture, we perform comparison between the full UMT model and its ablations with respect to the auxiliary entity span detection (ESD) module and the multimodal interaction (MMI) module.
As shown in Table 3, we can see that all the components in UMT make important contributions to the final results. On the one hand, removing the whole ESD module will significantly drop the performance, which shows the importance of alleviating the visual bias. In particular, discarding the conversion matrix in the ESD module also leads to the performance drop, which indicates the usefulness of capturing the label correspondence between the auxiliary module and our main MNER task.
On the other hand, as the main contribution of
3348
Importance of the MMI Module	Importance of the ESD Module	Importance of Associated Images	Noise of Associated Images
A. Review of [Wolf Hall MISC]1, Episode 1 : Three Card Trick (bit.ly/1BHnWNb) #[WolfHall MISC]2	B. [Kevin Love PER]1 was more ex-cited about [Gameof Thrones MISC]2 than beating the [Hawks ORG]3	C. My mum took some awesome photos of @ [iamrationale PER]1 and @ [bastilledan PER]2.	D. Ask [Siri MISC]1 what 0 divided by 0 is and watch her put you in your place.
BERT-CRF:	1-LOCX, 2-LOCX	1-PER✓, 2-MISC✓, 3-ORG✓	1-MISCX, 2-ORGX	1-MISC✓
AdaCAN-BERT-CRF:	1-LOCX, 2-LOCX	1-PER✓, 2-NONEX, 3-ORG✓	1-PER✓, 2-PER✓	1-PERX
MT-BERT-CRF:	1-MISC✓, 2-MISC✓	1-PER✓, 2-NONEX, 3-ORG✓	1-PER✓, 2-PER✓	1-PERX
UMT-BERT-CRF:	1-MISC✓, 2-MISC✓	1-PER✓, 2-MISC✓, 3-ORG✓	1-PER✓, 2-PER✓	1-PERX
Table 4: The second row shows several representative samples together with their manually labeled entities in the test set of our two TWITTER datasets, and the bottom four rows show predicted entities of different methods on these test samples.
our MMI module, Image-Aware Word Representations (WR) demonstrates its indispensable role in the final performance due to the moderate performance drop after removal. Besides, removing the visual gate also results in minor performance drop, indicating its importance to the full model.
3.5 Further Analysis
Importance of MMI and ESD Modules: To better appreciate the importance of two main contributions (i.e., MMI and ESD modules) in our proposed approaches, we conduct additional analysis on our two test sets. In Fig. 3 and Fig. 4, we show the number of entities that are wrongly/correctly predicted by BERT-CRF, but correctly/wrongly predicted by each multimodal method<sup>5</sup>.
First, we can see from Fig. 3 that with the MMI module, our MT-BERT-CRF and UMT-BERT-CRF approaches correctly identify more entities, compared with the two multimodal baselines. Table 4.A shows a specific example. We can see that our two methods correctly classify the type of Wolf Hall as MISC whereas the compared systems wrongly predict its type as LOC, probably because our MMI module enforces the image-aware word representations of Wolf Hall to be closer to drama names.
Second, in Fig. 4, it is clear to observe that compared with the other three methods, UMT-BERT-CRF can significantly decrease the bias brought by the visual context due to incorporating our auxiliary ESD module. In Table 4.B, we show a concrete example: since Game of Thrones is ignored by the image, the two multimodal baselines fail to identify them; in contrast, with the help of the auxiliary
ESD module, UMT-BERT-CRF successfully eliminates the bias.
Effect of Incorporating Images: To obtain a better understanding of the general effect of incorporating associated images into our MNER task, we carefully examine our test sets and choose two representative test samples to compare the prediction results of different approaches.
First, we observe that most improvements gained by multimodal methods come from those samples where the textual contents are informal or incomplete but the visual context provides useful clues. For example, in Table 4.C, we can see that without the visual context, BERT-CRF fails to identify that the two entities refer to two singers in the concert, but all the multimodal approaches can correctly classify their types after incorporating the image.
Second, by manually checking the test set of our two datasets, we find that in around  $5\%$  of the social media posts, the associated images might be irrelevant to the textual contents due to two kinds of reasons: (1) these posts contain image memes, cartoons, or photos with metaphor; (2) their images and textual contents reflect different aspects of the same event. In such cases, we observe that multimodal approaches generally perform worse than BERT-CRF. A specific example is given in Table 4.D, where all the multimodal methods wrongly classify Siri as PER because of the unrelated face in the image.
As a crucial component of many information extraction tasks including entity linking (Derczynski et al., 2015), opinion mining (Maynard et al., 2012), and event detection (Ritter et al., 2012), named
5Note that here we use strict matches (i.e., correct span and type predictions).
entity recognition (NER) has attracted much attention in the research community in the past two decades (Li et al., 2018).
Methods for NER: In the literature, various supervised learning approaches have been proposed for NER. Traditional approaches typically focus on designing various effective NER features, followed by feeding them to different linear classifiers such as maximum entropy, conditional random fields (CRFs), and support vector machines (Chieu and Ng, 2002; Florian et al., 2003; Finkel et al., 2005; Ratinov and Roth, 2009; Lin and Wu, 2009; Passos et al., 2014; Luo et al., 2015). To reduce the feature engineering efforts, a number of recent studies proposed to couple different neural network architectures with a CRF layer (Lafferty et al., 2001) for word-level predictions, including convolutional neural networks (Collobert et al., 2011), recurrent neural networks (Chiu and Nichols, 2016; Lample et al., 2016), and their hierarchical combinations (Ma and Hovy, 2016). These neural approaches have been shown to achieve the state-of-the-art performance on different benchmark datasets based on formal text (Yang et al., 2018).
However, when applying these approaches to social media text, most of them fail to achieve satisfactory results. To address this issue, many studies proposed to exploit external resources (e.g., shallow parser, Freebase dictionary, and orthographic characteristics) to incorporate a set of tweet-specific features into both traditional approaches (Ritter et al., 2011; Li et al., 2014; Baldwin et al., 2015) and recent neural approaches (Limsopatham and Collier, 2016; Lin et al., 2017), which can obtain much better performance on social media text.
Methods for Multimodal NER (MNER): As multimodal data become increasingly popular on social media platforms, several recent studies focus on the MNER task, where the goal is to leverage the associate images to better identify the named entities contained in the text. Specifically, Moon et al. (2018) proposed a multimodal NER network with modality attention to fuse the textual and visual information. To model the inter-modal interactions and filter out the noise in the visual context, Zhang et al. (2018) and Lu et al. (2018) respectively proposed an adaptive co-attention network and a gated visual attention mechanism for MNER. In this work, we follow this line of work. But different
from them, we aim to propose an effective multimodal method based on the recent Transformer architecture (Vaswani et al., 2017). To the best of our knowledge, this is the first work to apply Transformer to the task of MNER.
In this paper, we first presented a Multimodal Transformer architecture for the task of MNER, which captures the inter-modal interactions with a multimodal interaction module. Moreover, to alleviate the bias of the visual context, we further proposed a Unified Multimodal Transformer (UMT), which incorporates an entity span detection module to guide the final predictions for MNER. Experimental results show that our UMT approach can consistently achieve the best performance on two benchmark datasets.
There are several future directions for this work. On the one hand, despite bringing performance improvements over existing MNER methods, our UMT approach still fails to perform well on social media posts with unmatched text and images, as analyzed in Section 3.5. Therefore, our next step is to enhance UMT so as to dynamically filter out the potential noise from images. On the other hand, since the size of existing MNER datasets is relatively small, we plan to leverage the large amount of unlabeled social media posts in different platforms, and propose an effective framework to combine them with the small amount of annotated data to obtain a more robust MNER model.
We would like to thank three anonymous reviewers for their valuable comments. This research is supported by the National Research Foundation, Singapore under its International Research Centres in Singapore Funding Initiative, and the Natural Science Foundation of China under Grant 61672288. Any opinions, findings and conclusions or recommendations expressed in this material are those of the authors and do not reflect the views of National Research Foundation, Singapore.
Timothy Baldwin, Marie-Catherine de Marneffe, Bo Han, Young-Bum Kim, Alan Ritter, and Wei Xu. 2015. Shared tasks of the 2015 workshop on noisy user-generated text: Twitter lexical normalization and named entity recognition. In Proceedings of the Workshop on Noisy User-generated Text, pages 126-135.
Hai Leong Chieu and Hwee Tou Ng. 2002. Named entity recognition: a maximum entropy approach using global information. In Proceedings of COLING.
Jason PC Chiu and Eric Nichols. 2016. Named entity recognition with bidirectional lstm-cnns. Transactions of the Association for Computational Linguistics, 4:357-370.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research, 12:2461-2505.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255.
Leon Derczynski, Diana Maynard, Giuseppe Rizzo, Marieke Van Erp, Genevieve Gorrell, Raphael Troncy, Johann Petrak, and Kalina Bontcheva. 2015. Analysis of named entity recognition and linking for tweets. Information Processing & Management, 51(2):32-49.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of ACL.
Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. Named entity recognition through classifier combination. In Proceedings of NAACL.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of CVPR, pages 770-778.
Zhiheng Huang, Wei Xu, and Kai Yu. 2015. Bidirectional lstm-crf models for sequence tagging. arXiv preprint arXiv:1508.01991.
John Lafferty, Andrew McCallum, and Fernando CN Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of ICML.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. Neural architectures for named entity recognition. In Proceedings of NAACL-HLT.
Chenliang Li, Aixin Sun, Jianshu Weng, and Qi He. 2014. Tweet segmentation and its application to named entity recognition. IEEE Transactions on knowledge and data engineering, 27(2):558-570.
Chenliang Li, Jianshu Weng, Qi He, Yuxia Yao, Anwitan Datta, Aixin Sun, and Bu-Sung Lee. 2012. Twiner: named entity recognition in targeted twitter stream. In Proceedings of SIGIR.
Jing Li, Aixin Sun, Jianglei Han, and Chenliang Li. 2018. A survey on deep learning for named entity recognition. arXiv preprint arXiv:1812.09449.
Nut Limsopatham and Nigel Collier. 2016. Bidirectional LSTM for named entity recognition in twitter messages. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT).
Bill Yuchen Lin, Frank F Xu, Zhiyi Luo, and Kenny Zhu. 2017. Multi-channel bilstm-crf model for emerging named entity recognition in social media. In Proceedings of the 3rd Workshop on Noisy User-generated Text.
Dekang Lin and Xiaoyun Wu. 2009. Phrase clustering for discriminative learning. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Natural Language Processing of the AFNLP.
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In Proceedings of ACL, pages 1990-1999.
Gang Luo, Xiaojiang Huang, Chin-Yew Lin, and Za-iqing Nie. 2015. Joint entity recognition and disambiguation. In Proceedings of EMNLP.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional lstm-cnns-crf. In Proceedings of ACL.
Diana Maynard, Kalina Bontcheva, and Dominic Rout. 2012. Challenges in developing opinion mining tools for social media. In Proceedings of the @ NLP can u tag usergeneratedcontent.
Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity recognition for short social media posts. In Proceedings of NAACL.
Alexandre Passos, Vineet Kumar, and Andrew McCallum. 2014. Lexicon infused phrase embeddings for named entity resolution. In Proceedings of CoNLL.
Lev Ratinov and Dan Roth. 2009. Design challenges and misconceptions in named entity recognition. In Proceedings of CoNLL.
Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: an experimental study. In Proceedings of ACL.
Alan Ritter, Oren Etzioni, and Sam Clark. 2012. Open domain event extraction from twitter. In Proceedings of SIGKDD.
Erik F Sang and Jorn Veenstra. 1999. Representing text chunks. In Proceedings of EACL.
Kentaro Torisawa et al. 2007. Exploiting wikipedia as external knowledge for named entity recognition. In Proceedings of EMNLP-CoNLL.
Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. Multimodal transformer for unaligned multimodal language sequences. In Proceedings of ACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS, pages 5998-6008.
Vikas Yadav and Steven Bethard. 2018. A survey on recent advances in named entity recognition from deep learning models. In Proceedings of COLING.
Jie Yang, Shuailong Liang, and Yue Zhang. 2018. Design challenges and misconceptions in neural sequence labeling. In Proceedings of COLING.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In Proceedings of AAAI, pages 5674-5681.
GuoDong Zhou and Jian Su. 2002. Named entity recognition using an hmm-based chunk tagger. In Proceedings of ACL.