SlowGuess/NAACL_2022_OCR huggingface dataset:
Hao Hao Tan and M. Bansal. 2019. Lxmert: Learning cross-modality encoder representations from transformers. In EMNLP.
Erik F. Tjong Kim Sang. 2002. Introduction to the CoNLL-2002 shared task: Language-independent named entity recognition. In *COLING-02: The 6th Conference on Natural Language Learning* 2002 (CoNLL-2002).
Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142-147.
Alakananda Vempala and Daniel Preoziuc-Pietro. 2019. Categorizing and inferring the relationship between the text and image of Twitter posts. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2830-2840, Florence, Italy. Association for Computational Linguistics.
Xinyu Wang, Yong Jiang, Nguyen Bach, Tao Wang, Zhongqiang Huang, Fei Huang, and Kewei Tu. 2021. Improving named entity recognition by external context retrieving and cooperative learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 1800-1812, Online. Association for Computational Linguistics.
Yue Wang, Jing Li, Michael Lyu, and Irwin King. 2020. Cross-media keyphrase prediction: A unified framework with multi-modality multi-head attention and image wordings. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3311-3324, Online. Association for Computational Linguistics.
Qi Wu, Chunhua Shen, Lingqiao Liu, Anthony Dick, and Anton Van Den Hengel. 2016. What value do explicit high level concepts have in vision to language problems? In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 203-212.
Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-fung Leung, and Qing Li. 2020. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts. ACM MM.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. 2020. LUKE: Deep contextualized entity representations with entity-aware self-attention. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6442–6454, Online. Association for Computational Linguistics.
Peter Young, Alice Lai, M. Hodosh, and J. Hockenmaier. 2014. From image descriptions to visual denotations:
New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2:67-78.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. In AAAI.
Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. 2020. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3342-3352, Online. Association for Computational Linguistics.
Dong Zhang, Suzhong Wei, Shoushan Li, Hanqian Wu, Qiaoming Zhu, and Guodong Zhou. 2021a. Multi-modal graph fusion for named entity recognition with targeted visual guidance. In Proceedings of the AAAI Conference on Artificial Intelligence.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. 2021b. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5579-5588.
Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. 2018. Adaptive co-attention network for named entity recognition in tweets. In Thirty-Second AAAI Conference on Artificial Intelligence.
A.1 Details of Experiment Settings
We run our code on Tesla V100 GPU with 16 GB memory. It takes about two hours to train a model. The size of model parameter is approximately equal to size of BERT/XLMR embeddings.
A.2 Details of OCA
Table 6 shows that the OCR system only finds about  $26\%$  sentences have texts in the image and the extracted texts have an average of 28 tokens. The statistics show that ITA-OCA can help to improve the model accuracy with only  $26\%$  of the samples have OCR texts.
Despite that images can generally help to improve the accuracy of the NER model, there are a lot of cases that the images may contain misleading information to hurt the model prediction. We study two cases for LA nad GA: 1) the entities are wrongly predicted by BERT-CRF baseline but are correctly
predicted by ITA; 2) the entities are wrongly predicted by ITA without CVA but are correctly predicted by the baseline and ITA with CVA. Figure 4 shows the two cases with two samples for each. Figure 4 (a) shows the first case, which shows the importance of the visual contexts. The baseline model failed to recognize the person entities "TWICE" and "Harry Potter" possibly because the two words are usually an adverb and a book name respectively. For the  $\mathbf{I} + \mathbf{T}$  input view, our MNER model is able to recognize the hints such as "two girls", "young girl", "a couple of young men" and "woman" in the visual contexts and then correctly predict the two entities. Figure 4 (b) shows the second case, which shows how the noises from the image mislead the model predictions. There are three- and two-person entities in gold labels but the visual contexts indicate that the top right image has "two baseball players" and the bottom right image has only "a woman". As a result, ITA without CVA only predict two and one person entities according to the visual contexts in the two samples respectively. However, with CVA, ITA takes a good balance in utilizing the textual and visual information and correctly predicts the entity labels in both T and I+T input views.
For OCA, we study how the extracted texts can help model prediction. In the upper sample of Figure 5, there are two "Donald" words in the image. The baseline model failed to identify the latter one while ITA-OCA can successfully identify both of them. In the bottom of Figure 5, the texts in the image are mainly talking about "HARRY STYLES", which helps the model prediction.
In our paper, we use the captioning and object detection model based on MSCOCO and visual genome. The model performance could be improved if we use domain-specific models (Twitter domain). For OCA, the model accuracy may be poor if the OCR system does not support a certain language.
A.5 Loss Function Comparison with UMT
In the codes of UMT, the BERT embeddings tokenize the token in a sentence into subtokens. The codes use the first subtoken as the token representation to predict the corresponding label. However, for the other subtokens, the codes use a special label "PAD" for prediction. Therefore, the target labels are changed. For example, the original label
Figure 4: Examples of the positive and negative effects of images. The named entities in the text are colored. The wrongly predicted entities are marked in bold and colored in red. The missing entities are marked with  $\times$ . We use BIOES format to represent the label spans (https://en.wikipedia.org/wiki/Inside-outside-beginning_(tagging))
Figure 5: Examples of the positive effects of OCA. The named entities in the text are colored.
Twitter-15	Twitter-17	SNAP
Num Sents w/ OCR / Total Sents	2,049 / 8,288 (24.72%)	1,197 / 4,461 (26.83%)	1,869 / 7,181 (26.03%)
Avg. Length	27.72	27.00	28.93
Table 6: A statistic about the number of sentences has OCR texts and the average length of OCR texts.
sequence is "B-X, I-X, O, B-X, O, O" but now it becomes "B-X, PAD, PAD, I-X, O, B-X, O, PAD, O". As a result, the exact training objective changes compared with the training objective in the paper of UMT. We improve the code by removing all the "PAD" labels and just use the first substring of each token as the token representation. Our improved baseline model is significantly improved, while the accuracy of UMT model in the improved code cannot be further improved.