# Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction
## 1 Introduction  
Named Entity Recognition (NER) and Relation Extraction (RE) are core tasks in social media information monitoring and evaluation. Given the rising influence of multimodal posts, multimodal information extraction technologies are more conducive to the supervision of public opinion on social media, such as Multimodal Named Entity Recognition (MNER) and Multimodal Relation Extraction (MRE). In recent years, food safety issues have received widespread attention on social media, particularly regarding the safety of staple crops such as rice and wheat [1]. The spread and attention of food safety incidents typically affect consumer confidence and purchasing decisions. Therefore, monitoring and analyzing food safety information on social media, especially the safety information of staple crops, has become increasingly important [2]. To more effectively mine this information, researchers have applied NER and RE techniques to the monitoring and evaluation of food safety incidents, as well as the analysis of food industry dynamics and food recall information [3, 4]. On social media platforms, users comment, discuss, and share information about food safety issues, providing valuable data sources for researchers and regulatory authorities. In early NER and RE research, the focus was mainly on text modality studies. Traditional approaches mainly have focused on constructing various efficient NER features, which were subsequently fed into different linear classifiers, such as the Support Vector Machines, Logistic Regression, and Conditional Random Fields (CRFs) [5; 6; 7; 8; 9]. With the development of deep learning techniques, the traditional one-hot representations have been replaced by word embedding methods, such as Word2 Vec [10] and GloVe [11]. These methods capture semantic information of words in lower dimensions. In order to simplify the task of feature engineering, researchers have explored the combination of various deep neural networks with CRF layers to achieve word-level predictions--for instance, Recurrent Neural Network (RNN) [12] and Convolutional Neural Network (CNN) [13]. With the advent of the Transformer [14], many researchers adopted this model for NER and RE tasks [15; 16; 17]. Knowledge-enabled Bidirectional Encoder Representation from Transformers (K-bert) [18] and Enhanced Language Representation with Informative Entities (ERNIE) [19] combined pre-trained models with knowledge graphs to effectively learn additional entity knowledge. Although the above models have achieved good performance in the text modality, they still face challenges when dealing with short, noisy data, such as Twitter datasets, especially encountering out-of-vocabulary words and complex abbreviations, as this type of text is often ambiguous. Many researchers have also proposed coping strategies. [PERSON] et al. [20] employed sentence-level features to enhance the recognition accuracy of named entities in short texts. [PERSON] et al. [21] proposed a novel vocabulary substitution strategy based on constructing Entity Type Compatible (ETC) semantic spaces. Unknown words are replaced with ETC words found through Deep Metric Learning (DML) and Nearest Neighbor (NN) search. [PERSON] et al. [22] fine-tuned pre-trained models with blank relationship data to automatically determine whether entities were shared between two relationship instances. Language Understanding with Knowledge-based Embeddings (LUKE) [23] introduced an entity-aware self-attention strategy and contextualized representation. Nevertheless, the aforementioned methods have not exploited the potential of visual information when dealing with multimodal data. To compensate for the semantic context deficiencies in a single modality, [PERSON] et al. [24] constructed a deep image network and a generic modality attention module that leverages images to enhance textual information and diminish irrelevant modalities. The model aimed to uncover correlations between multimodal data. [PERSON] et al.