SlowGuess/NAACL_2022_OCR huggingface dataset:
In Table 2, we compare ITA with previous state-of-the-art approaches. For previous approaches, we report the results including OCSGA, UMT, RIVA, RpBERT, UMGF, which are the proposed approaches of Wu et al. (2020), Yu et al. (2020), Sun et al. (2020), Sun et al. (2021) and Zhang et al. (2021a) respectively. For fair comparison, we report the results of these models based on the BERT base embeddings. Moreover, since most of these previous approaches report the best model accuracy instead of the averaged model accuracy, we use the best model accuracy of ITA-All $_{+}$ CVA over 5 runs. We also report our reproduced results of UMT, RpBERT and UMGF on the corresponding datasets. The results show that ITA-All $_{+}$ CVA outperforms all of the previous approaches. On the SNAP dataset, the reported accuracy of RpBERT $_{\text{base}}$  is competitive with ITA-All $_{+}$ CVA. However, we find that the accuracy of our reproduced RpBERT $_{\text{base}}$  is significantly lower than the reported accuracy, even after careful check of the source code and hyper-parameter tuning. Moreover, the fact that our BERT-CRF baseline achieves competitive accuracy with previous state-of-the-art multi-modal approaches shows that most of the previous work has not fully explored the strength of the text representations for the task.
Approaches	Twitter-15	Twitter-17
BERT-CRFUMT	71.81	83.44
BERT-CRFOurs	74.79	85.18
OUR REPRODUCTIONS
BERT-CRFUMT	71.74	84.20
BERT-CRFUMT-Improved	72.53	84.48
UMT	72.83	84.88
UMTImproved	72.96	84.50
Table 3: Our reproductions of previous baselines and approaches. "Improved" means our improved models based on the UMT code base.
Discussion about Textual Modules As we have shown in Table 1 and 2, the textual baselines (i.e. BERT-CRF) of previous work are significantly lower than that of ours. In most of the previous MNER architectures, the textual modules are mainly based on the baseline architectures with some modifications. We further show the baselines of previous work are not well-trained and how the multi-modal approaches perform with stronger textual modules. In Table 3, we rerun the BERT-CRF baseline based on the released codes of UMT<sup>8</sup>. Based on the code of UMT, we tried to improve the baseline models in the code by using the same loss function as ours<sup>9</sup>. The accuracy of BERT-CRF models in the code are significantly improved but the UMT models based on the improved code are not improved and even get worse in Twitter-17. Therefore, we suspect the UMT model cannot be further improved even with stronger textual modules. Zhang et al. (2021a) also reported the baseline based on the implementation of Yu et al. (2020), so we suspect the UMGF model cannot be improved as well. Therefore, the under-trained textual baselines of previous work make the effectiveness of the images unclear and we show that some of the MNER models perform even weaker than our BERT-CRF model.
3.3 Comparison with Other Variants
To further show the effectiveness of ITA, we perform several comparisons between ITA and the following variants of the MNER model in Table 4:
ITA-Random: We generate random image-text pairs for the model. For each sentence, we randomly select the image in the dataset and generate the corresponding visual contexts. The noises of random visual contexts make the model accuracy
<sup>7</sup>We reproduced the results based on the official code for RpBERT<sub>base</sub>: https://github.com/Multimodal-NER/RpBERT
$^{8}$ https://github.com/jefferyu/UMT
9The details are discussed in Appendix A.5
Twitter-15	Twitter-17	SNAP
Approach	Eval Modal	Eval Modal	Eval Modal
T	I+T	T	I+T	T	I+T
ITA-Random	-	74.67	-	84.98	-	85.82
ITA-GABU	-	75.10	-	85.77	-	86.51
ITA-LABU	-	75.18	-	85.59	-
ITA-OCAPaddle	-	75.12	-	85.87	-	86.66
BERT-CRF+ImgFeat	-	74.70	-	84.99	-	85.90
VinVL-CRF	-	60.58	-	75.55	-	74.53
BERT+VinVL-CRF	-	74.89	-	85.19	-	86.14
ITA-Joint	74.88	75.22	85.31	85.60	86.06	86.34
REFERENCES
RpBERT w/o Rp	-	72.60	-	-	-	86.20
ITA-All+CVA	75.50	75.41	85.89	85.84	86.83	86.75
Table 4: A comparison of other variants of MNER models.
drop slightly comparing with our BERT-CRF baseline, which shows the improvement of our approach is from the visual contexts rather than extending the input sequence length the embeddings.
ITA-Joint: It is an ablated model of ITA-All $_{+}$ CVA. We train the ITA-All model for both input views without the CVA loss in Eq. 5. The model accuracy is improved moderately with only the T input view while our ITA-All $_{+}$ CVA can improve both input views significantly, which shows the effectiveness of the CVA module of ITA.
ITA-LABU and ITA-GABU: We conduct experiments to see how the accuracy changes when using weaker image features. We use Bottom-Up features proposed by Anderson et al. (2018) for object detection and image captioning. The captioning model is a pretrained image captioning model<sup>10</sup> proposed by Luo et al. (2018) with the Bottom-Up features and self-critical training (Rennie et al., 2017). Results show that there is no significant difference between the visual contexts from Bottom-Up features and VinVL features. Therefore, our approaches can utilize other off-the-shelf vision models to extract visual contexts.
ITA-OCAPaddle: We conduct experiments to see how the accuracy changes when using stronger OCR models. We use PaddleOCR<sup>11</sup> for the experiment, which is one of the newest open resource lightweight OCR system. Results show that the model accuracy can be slightly improved comparing with ITA-OCA, which shows the ITA models
Figure 2: A relation between the number of captions input to the MNER model and model accuracy. The x-axis is the number of captions. The y-axis is the averaged F1 score on the test set.
can be improved by using better OCR models.
BERT-CRF $_{+\text{ImgFeat}}$ : Instead of ITA, we can directly feed the image region features generated from an object detector into the BERT. We use ResNet-152 model to generate region features and then feed the features into a linear layer to project the region features into the same space of text features in the BERT. Moreover, we compare the model with RpBERT w/o Rp, which is an ablated model of RpBERT and is equivalent to BERT-CRF $_{+\text{ImgFeat}}$  over the usage of BERT embeddings. Sun et al. (2021) showed RpBERT w/o Rp can improve the model accuracy compared with their baseline. However, our results show that the model accuracy slightly drops comparing with our BERT-CRF, which shows that it is difficult for the attention module of BERT to learn the relations of the unaligned representations of two modalities.
VinVL-CRF: To show how the pretrained  $\mathrm{V + L}$  models perform on the NER task, we use VinVL since it is a very recent state-of-the-art pretrained  $\mathrm{V + L}$  model on a lot of multi-modal tasks. We feed the VinVL model with texts and images in the MNER datasets and finetune the model over the task. We take the text representations output from VinVL as the input of the CRF layer. The accuracy of the finetuned VinVL model drops significantly compared to the BERT model, which shows that the inductive bias of the pretrained  $\mathrm{V + L}$  model hurts the model accuracy on MNER.
BERT+VinVL-CRF: As the VinVL model may lead to an inductive bias over the common nouns and the image, we jointly finetune the BERT and VinVL models and concatenate the output text representations of the two models. The accuracy is improved on a moderate scale, which shows BERT is complementary to VinVL for MNER.
10https://github.com/ruotianluo/   
self-critical.pytorch   
11https://github.com/PaddlePaddle/   
PaddleOCR
Figure 3: Averaged L2 distance between the token representations without image input  $(\boldsymbol{r}_i)$  and with image input  $(\boldsymbol{r}_i^{\prime})$ . The error bars mean the standard deviation over 5 runs.
Effect of the Number of Captions Using more captions output from the captioning model can improve diversities of the visual contexts but can add noises to them as well. To better understand how the number of captions affects the model accuracy, we change the beam size and keep all the sentences output from the captioning model. The trends in Figure 2 show that the model accuracy increases until 5 captions for all the datasets and gradually drops when the number of captions further increases for Twitter-15 and 17 datasets. The observation shows that using 5 captions keeps a good balance between the diversities and correctness of the captions.
How ITA Eases the Cross-Modal Alignments
Previous work such as Moon et al. (2018); Sun et al. (2021) visualized modality attention in several cases to show the effectiveness of their approaches. However, visualizing the multi-layer attention in transformer-based embeddings is relatively difficult. Instead of studying special cases, we statistically calculate the averaged L2 distance between token representations  $r_i$  and  $r_i'$  from two input modalities to show how the token representations depend on image information. In Figure 3, the L2 distance ITA-All is significantly larger than that of BERT-CRF+ImgFeat. Besides, the standard deviation of BERT-CRF+ImgFeat is very large. The observations show the image region features make the alignment become difficult and unstable while our visual contexts can significantly ease the cross-modal alignments. Moreover, with CVA, the L2 distance becomes much smaller and stable as CVA aligns the two input views to reduce the dependence on images, which shows the MNER model can better utilize the textual information with CVA.
How Images Affect the NER Prediction To study the effectiveness of the images over each
label, we show a comparison between our model and our baselines in Table 5. When the relative improvement of the F1 score is larger than 0.5, the relative improvement of precision is larger than that of recall. The observation shows that the main improvement of MNER is mainly because the images can help the model to reduce false-positive predictions for disambiguation on uncertain entities.[12]
Multi-modal Named Entity Recognition Most of the previous approaches to MNER focus on the interaction between image and text features through attention mechanisms. Moon et al. (2018) proposed a modality attention network to fuse the text and image features before the input to the BiLSTM layer. Lu et al. (2018) additionally used a visual attention gate for the output features of the BiLSTM layer. Zhang et al. (2018) proposed an adaptive co-attention network after the BiLSTM layer to model the interaction between image and text. Recently, Wu et al. (2020) proposed OCSGA, which use object labels to model the interaction between image and object labels in an additional dense co-attention layer. Compared with the work, we show a simpler and more effective way to utilize object labels and additionally use other alignment approaches to further improve the model accuracy. Yu et al. (2020) proposed UMT, which utilized a multi-modal interaction module and an auxiliary entity span detection module for MNER. Zhang et al. (2021a) proposed UMGF, which utilizes a pretrained parser to create the graph connection between visual object tags and textual words. They used a graph attention network to fuse the textual and visual features. In order to better model whether the image is related to the text, Sun et al. (2021) proposed RpBERT, which additionally trains on a text-image relation classification dataset proposed by Vempala and Preoziuc-Pietro (2019) to prevent the negative effect of noisy images. Comparing with RpBERT, we use CVA to let the NER model better utilize the input sentences without such kinds of supervision. All of these approaches focus on fusing the image and text features through the attention mechanism but ignore the gap between the image and text features while we propose to fully utilize the attention mechanism in the pretrained textual embeddings through
$^{12}$ In Appendix A.3, we show several cases to show the effectiveness of ITA to affect NER prediction.
LOC	ORG	PER	OTHER
P	R	F1	P	R	F1	P	R	F1	P	R	F1
Twitter-15
BERT-CRF	80.0	83.8	81.8	65.9	61.0	63.3	84.2	86.8	85.4	44.2	44.2	44.1
ITA-All+CVA	81.1	84.2	82.6	68.8	60.6	64.4	84.0	87.2	85.6	44.9	44.6	44.8
Δ	1.1	0.4	0.8	2.8	-0.4	1.1	-0.2	0.4	0.1	0.8	0.5	0.6
Twitter-17
BERT-CRF	85.5	84.4	84.9	83.5	83.8	83.7	90.7	90.8	90.7	68.9	65.1	66.9
ITA-All+CVA	86.0	83.7	84.8	83.9	84.2	84.0	91.9	90.9	91.4	73.7	64.3	68.6
Δ	0.5	-0.7	-0.1	0.3	0.4	0.4	1.2	0.1	0.7	4.8	-0.8	1.7
SNAP
BERT-CRF	82.1	82.8	82.5	87.8	86.9	87.3	91.0	91.5	91.2	72.3	75.1	73.7
ITA-All+CVA	80.3	81.7	81.0	87.8	86.5	87.1	90.1	91.2	90.6	70.1	73.2	71.6
Δ	1.9	1.1	1.5	0.6	0.5	0.5	0.9	0.3	0.6	2.2	1.9	2.1
Table 5: A comparison between our ITA (ITA-All+CVA with I+T inputs) model and the baseline (BERT-CRF) in precision (P), recall (R) and F1.  $\triangle$  represents the relevant improvement of ITA over the Baseline.
aligning image features into textual space. Besides, some cross-media research also shows the effectiveness of OCR texts (Chen et al., 2016; Wang et al., 2020) and object tags (Wu et al., 2016) have been shown. Most of the approaches introduced a new attention module over cross-modal features while in comparison ITA effectively utilizes the attention module in the pretrained textual embeddings.
Pretrained Vision-Language Models Inspired by related work on language model pretraining, visual-language pretraining (VLP) has recently attracted a lot of attention (Li et al., 2019; Lu et al., 2019; Chen et al., 2020; Tan and Bansal, 2019; Li et al., 2020a; Yu et al., 2021; Zhang et al., 2021b). The pretrained  $\mathrm{V + L}$  models are pretrained on large-scale image-text pairs and have achieved state-of-the-art accuracy over various vision-language tasks such as image captioning, VQA, NLVR and imagedtext retrieval. Recently, Li et al. (2020a) proposed Oscar to add object tags in pretraining so that self-attention can learn the image-text alignments easily. Following Oscar, Zhang et al. (2021b) proposed VinVL to train a large-scale object detector to improve the pretrained  $\mathrm{V + L}$  model's accuracy. Comparing with VLP, MNER is a totally different task. Firstly, the image-caption pairs are given in VLP and the image and text are equally important in pretraining for general representations. Therefore, using global alignment is meaningless for VLP but makes sense for MNER. In MNER, the input text is not the caption of the image and the image may not adds additional information to the input text. Secondly, though captions and object tags are often utilized in VLP, how to effectively utilize the captions and object tags of the image in MNER is rarely considered. Finally, besides the local and
global alignments, another aspect of ITA is the optical character alignment and cross-view alignment, which is rarely considered in VLP.
In this paper, we propose Image-Text Alignments for multi-modal named entity recognition, which convert images into object labels, captions and OCR texts to align the image representations into textual space in a multi-level manner and form a cross-modal input view. The model can effectively utilize attention module of the transformer-based embeddings. Considering noises, availability of images and inference speed for practical use, we propose cross-view alignment, which let the MNER models better utilize the text information in the input. In our experiments, we show that ITA significantly outperforms previous state-of-the-art approaches on MNER datasets. We also show that most of the previous work failed to train a good textual baseline while our textual baseline can easily match or even outperform previous multi-modal approaches. In analysis, we further analyze how ITA eases the cross-modal alignments and how the images affect the NER prediction.
This work was supported by the National Natural Science Foundation of China (61976139) and by Alibaba Group through Alibaba Innovative Research Program.
Aishwarya Agrawal, Jiasen Lu, Stanislaw Antol, Margaret Mitchell, C. L. Zitnick, Devi Parikh, and Dhruv
Batra. 2015. Vqa: Visual question answering. International Journal of Computer Vision, 123:4-31.
Alan Akbik, Tanja Bergmann, and Roland Vollgraf. 2019. Pooled contextualized embeddings for named entity recognition. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 724-728, Minneapolis, Minnesota. Association for Computational Linguistics.
Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. Contextual string embeddings for sequence labeling. In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638-1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. 2018. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR.
Tao Chen, Xiangnan He, and Min-Yen Kan. 2016. Context-aware image tweet modelling and recommendation. In Proceedings of the 24th ACM international conference on Multimedia, pages 1018-1027.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020. Uniter: Universal image-text representation learning. In European conference on computer vision.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440-8451, Online. Association for Computational Linguistics.
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the WNUT2017 shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User-generated Text, pages 140-147, Copenhagen, Denmark. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Rezarta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. Journal of biomedical informatics, 47:1-10.
Besnik Fetahu, Anjie Fang, Oleg Rokhlenko, and Shervin Malmasi. 2021. Gazetteer enhanced named entity recognition for code-mixed web queries. In SIGIR '21, SIGIR '21, New York, NY, USA. Association for Computing Machinery.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778.
Zhiheng Huang, W. Xu, and Kaijiang Yu. 2015. Bidirectional lstm-crf models for sequence tagging. ArXiv, abs/1508.01991.
John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional random fields: Probabilistic models for segmenting and labeling sequence data. In Proceedings of the Eighteenth International Conference on Machine Learning, ICML '01, page 282-289, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Jiao Li, Yueping Sun, Robin J Johnson, Daniela Sciaky, Chih-Hsuan Wei, Robert Leaman, Allan Peter Davis, Carolyn J Mattingly, Thomas C Wiegers, and Zhiyong Lu. 2016. Biocreative v cdr task corpus: a resource for chemical disease relation extraction. Database: The Journal of Biological Databases and Curation, 2016.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. arXiv preprint arXiv:1908.03557.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. 2020a. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. 2020b. Oscar: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pages 121-137. Springer.
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),
pages 1990-1999, Melbourne, Australia. Association for Computational Linguistics.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS.
R. Luo, Brian L. Price, Scott D. Cohen, and Gregory Shakhnarovich. 2018. Discriminability objective for training descriptive captions. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6964-6974.
Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity recognition for short social media posts. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 852-860, New Orleans, Louisiana. Association for Computational Linguistics.
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7008-7024.
Stefan Schweter and Alan Akbik. 2020. Flert: Document-level features for named entity recognition. arXiv preprint arXiv:2011.06993.
Ray Smith. 2007. An overview of the tesseractOCR engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), volume 2, pages 629-633. IEEE.
Benjamin Strauss, Bethany Toma, Alan Ritter, Marie-Catherine de Marneffe, and Wei Xu. 2016. Results of the WNUT16 named entity recognition shared task. In Proceedings of the 2nd Workshop on Noisy User-generated Text (WNUT), pages 138-144, Osaka, Japan. The COLING 2016 Organizing Committee.
Alane Suhr, Stephanie Zhou, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In ACL.
Lin Sun, Jiquan Wang, Yindu Su, Fangsheng Weng, Yuxuan Sun, Zengwei Zheng, and Yuanyi Chen. 2020. RIVA: A pre-trained tweet multimodal model based on text-image relation for multimodal NER. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1852-1862, Barcelona, Spain (Online). International Committee on Computational Linguistics.
Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fangsheng Weng. 2021. Rpbert: A text-image relation propagation-based bert model for multimodal ner. In AAAI.
Beth M. Sundheim. 1995. Named entity task definition, version 2.1. In Proceedings of the Sixth Message Understanding Conference, pages 319-332.