SlowGuess/NAACL_2022_OCR huggingface dataset:
Firstly, we can find that incorporating the visual features is generally helpful for NER and RE tasks by comparing the SOTA multimodal approaches with their corresponding text-based baselines. Despite previous multimodal approaches can generally achieve better performance, the enormous improvement of F1 score for NER is only about  $2.0\%$  (compare UMGF with BERT-CRF), which for RE is about  $5.55\%$  (compare MEGA with MTB). This observation reveals that the performance improve
Figure 3: Performances on low-resource setting on MNER and MRE task.
ment of images on text-based NER tasks is relatively limited compared with RE tasks.
Secondly, our method is superior to the newest SOTA models UMGF and MEGA, which improves  $1.36\%$ , and  $15.44\%$  F1 scores for Twitter-2017, and MNRE datasets, respectively. It is worth noting that most of previous multimodal methods ignore the error sensitivity of irrelevant object-level images, while our method regard hierarchical visual prefix as a prompt for text. This results indicate that our method can effectively alleviate the error sensitivity irrelevant object images, which is a more robust method for visual enhanced NER and RE.
Finally, we also compare with VisualBERT, which is a pre-trained multimodal BERT with a single-stream structure. We notice that even as the pre-trained multimodal model, VisualBERT leaves much to be desired in MNER and MRE tasks, which performs worse than UMGF and MEGA, let alone our methods. We hold that VisualBERT is truly dissatisfactory since the datasets and pretraining process are less relevant to information extraction tasks.
Relevant Image-text Pair	Weak Relevant Image-text Pair	Irrelevant Image-text Pair
Taylor Hill holding Jun's GQ japan lol.	Cold front over Blyde River Canyon in Limpopo Province, South Africa.	President Bush when he sees the lights of America.
Text-Images Attention of HVPNeT
Taylor Hill Jun	Limpopo Province Blyde River Canyon	Bush America

Gold Relations: per/per/couple	loc/loc/contain	per/loc/place_of_residence
BERT: VisualBERT: MEGA: HVPNeT(Ours):	per / per /couple √	misc / misc /part_of √	per / loc/place_of_residence √
per / per /peer √	misc / misc /part_of √	misc / loc/held_on √
per / per /peer √	per / per /peer √	misc / loc/held_on √
per / per /peer √	loc / loc /contain √	per / loc/place_of_residence √
4.3.2 Low-resource Scenario
We further conduct experiments in low-resource settings by randomly sampling  $5\%$  to  $50\%$  from the original training set to form a low-resource training set. Figure 3 shows the performance of our method in a low-resource scenario compared with several baselines. By analyzing this results, we can observe: 1) UMT and MEGA consistently outperform the compared baselines in the low-resource scenario; the improvement indicates that incorporating the visual features is still helpful for NER and RE tasks in low-resource scenarios. 2) Moreover, it can be observed that the performance of HVPNeT still outperforms the other baselines. It further proves the effectiveness and data-efficiency of our proposed method.
4.3.3 Cross-task Scenario
Table 3 shows performance comparison of HVPNeT and UMGF in a cross-task scenario for versatility analysis. For the first part, Twitter  $2017 \rightarrow$  MNRE denotes that the trained model on Twitter-2017 is further used to train and test on MNRE. For the second part, MNRE  $\rightarrow$  Twitter-2017 represents that the trained model on Twitter-2017 is used to further train and test on Twitter-2017. From this Table, we can observe that our HVPNeT significantly outperforms UMGF by a more considerable margin. Note that our method can achieve further improvement in a cross-task scenario, while UMGF performs worse than previous results on the corresponding dataset. This justifies that our HVPNeT is robust to automatically reduce the interference of visual information of irrelevant pictures; thus,
Table 2: The first row shows the split of the relevance of image-text pairs, and the several middle rows indicate representative samples together with their entity-object attention in the test set of MNRE datasets (The y-axis represents the textual entities, and the x-axis denotes the visual objects with length of flattened 4 patches), and the bottom four rows show predicted relation of different approaches on these test samples.
Methods	Twitter-2017 → MNRE	MNRE → Twitter-2017
UMGF	63.85 → 62.90 ↓ (0.95)	85.51 → 84.35 ↓ (1.16)
HVPNeT	81.85 → 82.50 ↑ (0.75)	86.87 → 87.13 ↑ (0.26)
Table 3: Performance comparison of HVPNeT and UMGF in cross-task scenario.
more image-text data may facilitate learning better parameters for modality fusion. Besides, it is also interesting to extend our work to multi-task learning or multi-modal pre-training and we leave these for future works.
4.4 Detailed Model Analysis
Ablation Study. In this part, we conduct extensive experiments with the variants of our model to further analyze the effectiveness of our model. We illustrate the results of the variant set in Table 1. We can observe that:
(1) Visual Prefix-guided Fusion. The core module of our HVPNeT is visual prefix-guided fusion, which is a pluggable operation. Therefore, ablating visual prefix-guided fusion is equivalent to a purely bert-based baseline model. As shown in Table 1, HVPNeT achieve significant improvements over purely bert-based baseline model, revealing the effectiveness of pluggable visual prefix-guided fusion.
(2) Hierarchical Visual Features. To validate the impact of our proposed hierarchical visual features, we carry out experiments by introducing two variants: 1) HVPNeT-Flat, crudely assign single visual feature for each layer of BERT; and 2) HVPNeT-1T3, intuitively leveraging visual fea
Figure 4: Visualization of dynamic gate learned on MNER task. Each subgraph denotes one layer in BERT, and the ordinate and abscissa respectively represent the instance id in a batch and the block id of ResNet.
tures from low-level to high-level blocks. We observe that HVPNeT with hierarchical visual features achieves the best performance consistently compared with the other variants. Although the HVPNeT-1T3 performs slightly lower than the version of dynamic gate, it still outperforms the crude variant HVPNeT-Flat. It reveals that the dynamic gate can automatically learn appropriate weights for multi-scaled visual representations, enabling the model to learn good visual guidance for multimodal entity and relation extraction.
(3) Visual Clues Term. As recent SOTA models such as UMT, UMGF, and MEGA all adopt visual objects to enhance textual representation, we conduct experiments by ablating global images to explore the impact of the visual clues. As expected, we find that HVPNeT-OnlyObj performs slightly worse than HVPNeT, which is consistent with the observation of previous works. This can be attributed to that abstract clues maybe not be associated with the text in information extraction tasks. In other words, this empirical finding demonstrates the flexibility of our methods to infuse visual clues with different granularity.
Case Analysis for Error Sensitivity To validate the effectiveness and robustness of our method, we conduct case analysis for image-text relevance as indicated in Table 2. We notice that VisualBERT, MEGA, and our method can recognize the relation for the relevant image-text pair. We can further find that the attention between relevant entities and objects is significant. While in the situation that image represents the abstract semantic that is weak relevant to the text, only our method success in prediction due to HVPNeT captures the more hierarchical features. It should be noted that another two multimodal baselines fail in irrelevant image-text pairs while text-based BERT and ours still predict correctly. These observations reveal that our model regards visual prefix as a prompt
for text may helps learn more robust multimodal representation, which is essential for the noise of uncorrelated object images.
Gate Visualization We argue that dynamic gated aggregation for hierarchical visual representation is another key component of HVPNeT achieving the superior performance. Specifically, the dynamic gated aggregation can adaptively assign different modality integration paths for different input images, thus, incorporating visual guidance with hierarchical multi-scaled information. To this end, we randomly sample eight images in a batch and visualize their gate vectors learned by HVPNeT according to 12 layers of BERT in Figure 4. Note that optimized gate vectors follow the trend of matching low-level textual semantics with low-level visual semantics and matching high-level textual semantics with high-level visual semantics. Meanwhile, the modality fusion obtained by dynamic gate learning may provide some valuable insights for efficient visual-language approaches in the future.
In this paper, we propose a novel hierarchical visual prefix fusion neTwork (HVPNeT) for visual-enhanced entity and relation extraction. To be specific, we present visual prefix-guided fusion by concatenating object-level visual representation as the prefix of each self-attention layer in BERT, which is a more soft and robust attention module for visual enhanced NER and RE. We further design leveraging hierarchical multi-scaled visual representation as visual guidance for fusion. Intuitively, Good Visual Guidance Make A Better Extractor, and extensive experimental and results on three benchmarks have demonstrated the effectiveness and robustness of our proposed method. Meanwhile, our method also face the limitation that they don't suitable for multimodal tasks in visual side, such as visual grounding.
In the future, we plan to 1) explore more applications of hierarchical visual prefix in multimodal representation learning, making it more flexible and extensible; 2) try to apply the reverse version of our approach to boost visual representation with text for CV; 3) extend our approach to multitask multimodal pre-training.
We want to express gratitude to the anonymous reviewers for their hard work and kind comments. This work is funded by National Key R&D Program of China (Funding No.SQ2018YFC000004), NSFC91846204/NSFCU19B2027, Zhejiang Provincial Natural Science Foundation of China (No. LGG22F030011), Ningbo Natural Science Foundation (2021J190), and Yongjiang Talent Introduction Programme (2021A-156-G).
Omer Arshad, Ignazio Gallo, Shah Nawaz, and Alessandro Calefati. 2019. Aiding intra-text representations with visual context for multimodal named entity recognition. *ArXiv preprint*, abs/1904.01356.
Shuguang Chen, Gustavo Aguilar, Leonardo Neves, and Thamar Solorio. 2020a. A caption is worth A thousand images: Investigating image captions for multimodal named entity recognition. CoRR, abs/2010.12712.
Xiang Chen, Ningyu Zhang, Lei Li, Xin Xie, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, and Hua-jun Chen. 2021a. Lightner: A lightweight generative framework with prompt-guided attention for low-resource NER. CoRR, abs/2109.00720.
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. 2021b. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. CoRR, abs/2104.07650.
Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dolkar, and C. Lawrence Zitnick. 2015. Microsoft COCO captions: Data collection and evaluation server. CoRR, abs/1504.00325.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2020b. UNITER: universal image-text representation learning. In Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXX, volume 12375 of Lecture Notes in Computer Science, pages 104-120. Springer.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 3816-3830. Association for Computational Linguistics.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Hawre Hosseini. 2019. Implicit entity recognition, classification and linking in tweets. In Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2019, Paris, France, July 21-25, 2019, page 1448. ACM.
Ganesh Jawahar, Benoit Sagot, and Djamé Seddah. 2019. What does BERT learn about the structure of language? In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28-August 2, 2019, Volume 1: Long Papers, pages 3651-3657. Association for Computational Linguistics.
Seung-Wook Kim, Hyong-Keun Kook, Jee-Young Sun, Mun-Cheon Kang, and Sung-Jea Ko. 2018. Parallel feature pyramid network for object detection. In Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part V, volume 11209 of Lecture Notes in Computer Science, pages 239-256. Springer.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016a. Neural architectures for named entity recognition. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 260-270, San Diego, California. Association for Computational Linguistics.
Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016b. Neural architectures for named entity recognition. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
San Diego California, USA, June 12-17, 2016, pages 260-270. The Association for Computational Linguistics.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. 2020. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 11336-11344. AAAI Press.
Lianian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A simple and performant baseline for vision and language. ArXiv preprint, abs/1908.03557.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4582-4597. Association for Computational Linguistics.
Xiaozhuan Liang, Ningyu Zhang, Siyuan Cheng, Zhen Bi, Zhenru Zhang, Chuanqi Tan, Songfang Huang, Fei Huang, and Huajun Chen. 2022. Contrastive demonstration tuning for pre-trained language models. CoRR, abs/2204.04392.
Tsung-Yi Lin, Piotr Dólár, Ross B. Girshick, Kaiming He, Bharath Hariharan, and Serge J. Belongie. 2017. Feature pyramid networks for object detection. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 936-944. IEEE Computer Society.
Kun Liu, Yao Fu, Chuanqi Tan, Mosha Chen, Ningyu Zhang, Songfang Huang, and Sheng Gao. 2021. Noisy-labeled NER with confidence estimation. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 3437-3445. Association for Computational Linguistics.
Shu Liu, Lu Qi, Haifang Qin, Jianping Shi, and Jiaya Jia. 2018. Path aggregation network for instance segmentation. In 2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018, pages 8759-8768. Computer Vision Foundation / IEEE Computer Society.
Yijin Liu, Fandong Meng, Jinchao Zhang, Jinan Xu, Yufeng Chen, and Jie Zhou. 2019. GCDT: A global
context enhanced deep transition architecture for sequence labeling. In Proceedings of ACL, pages 2431-2441, Florence, Italy. Association for Computational Linguistics.
Di Lu, Leonardo Neves, Vitor Carvalho, Ning Zhang, and Heng Ji. 2018. Visual attention model for name tagging in multimodal social media. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1990-1999, Melbourne, Australia. Association for Computational Linguistics.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 13-23.
Xuezhe Ma and Eduard H. Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Seungwhan Moon, Leonardo Neves, and Vitor Carvalho. 2018. Multimodal named entity recognition for short social media posts. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 852-860, New Orleans, Louisiana. Association for Computational Linguistics.
Yujia Qin, Yankai Lin, Ryuichi Takanobu, Zhiyuan Liu, Peng Li, Heng Ji, Minlie Huang, Maosong Sun, and Jie Zhou. 2021. ERICA: improving entity and relation understanding for pre-trained language models via contrastive learning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 3350-3363. Association for Computational Linguistics.
Shaoqing Ren, Kaiming He, Ross B. Girshick, and Jian Sun. 2015. Faster R-CNN: towards real-time object detection with region proposal networks. In Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada, pages 91-99.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. 2018. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 2556-2565. Association for Computational Linguistics.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. 2019. Matching the blanks: Distributional similarity for relation learning. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 2895-2905. Association for Computational Linguistics.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2020. VL-BERT: pretraining of generic visual-linguistic representations. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Lin Sun, Jiquan Wang, Kai Zhang, Yindu Su, and Fangsheng Weng. 2021. Rpbert: A text-image relation propagation-based BERT model for multimodal NER. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Thirty-Third Conference on Innovative Applications of Artificial Intelligence, IAAI 2021, The Eleventh Symposium on Educational Advances in Artificial Intelligence, EAAI 2021, Virtual Event, February 2-9, 2021, pages 13860-13868. AAAI Press.
Hao Tan and Mohit Bansal. 2019. LXMERT: Learning cross-modality encoder representations from transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5100-5111, Hong Kong, China. Association for Computational Linguistics.
Kaihua Tang, Yulei Niu, Jianqiang Huang, Jiaxin Shi, and Hanwang Zhang. 2020. Unbiased scene graph generation from biased training. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 3713-3722. Computer Vision Foundation / IEEE.
Tiancai Wang, Rao Muhammad Anwer, Hisham Cholakkal, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao. 2019. Learning rich features at high-speed for single-shot object detection. In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 1971-1980. IEEE.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. 2021. Simvlm: Simple visual language model pretraining with weak supervision. CoRR, abs/2108.10904.
Zhiwei Wu, Changmeng Zheng, Yi Cai, Junying Chen, Ho-Fung Leung, and Qing Li 0001. 2020. Multimodal representation with embedded visual guiding objects for named entity recognition in social media posts. In MM '20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, pages 1038-1046. ACM.