# A Survey of Information Extraction Based on Deep Learning
## 4 Multi-Model IE Based on Deep Learning
### Multi-Modal Entity Identification  
Traditional named entity recognition only considers text information and ignores the influence of integrating other modes on the recognition of named entities. In view of the deficiency of using unimodal information to identify named entities, scholars have begun to study the task of named entity recognition combined with multi-modal information. Most multi-modal methods use the attention mechanism to extract visual information but ignore whether there a correlation exists between the text and the images, and text-independent visual information can have uncertain and even negative effects on the learning of multi-modal models. [PERSON] et al. proposed a multi-modal BERT model based on a text-image relationship propagation-based multimodal BERT model (RP-BERT) for text-image relationship classification (TRC), and trained the model RP-BERT on the MNER. The model achieved the highest F1 scores for both the TRC and MNER. Experimental results showed that the propagation of text-image relationships was able to reduce the interference of irrelevant images, and RP-BERT makes better use of visual information based on the text image relationship [97]. [PERSON] et al. constructed and obtained a large-scale labeled dataset containing multi-modal tweets from Twitter. To use visual information to identify named entities in multi-modal tweets, [PERSON] proposed an adaptive co-attention network (ACN) linking text and visual information (as shown in Figure 10) [98]. An adaptive co-attention network layer was inserted between the hidden and CRF layers to pay mutual attention to the representation of text and pictures; thus, each word acquired a multi-modal representation by introducing a gated multi-modal fusion module to decide when to rely on visual information. A filter gate module was also used to filter the noise caused by visual information. The model introduced image information based on CNN + BiLSTM + CRF, adding an ACN module on the constructed dataset with 72.75% accuracy, recall of 68.74%, and 70.79% F1, with better performance than CNN + BiLSTM + CRF. Some MNER models do not take full advantage of the fine-grained semantic correspondence between different modal semantic units, which may optimize multi-modal representation learning. A unified multi-modal graph-fusion (UMGF) approach for MNER was proposed by [PERSON] et al. [99]. A unified multi-modal graph was first used to represent the input sentences and images. During the composition process, each target image acts as an image node. Each word acted as a text node. The graph captured the various semantic relationships between modal semantic units (words and visual objects). After stacking multiple graph-based multi-modal fusion layers, semantic interactions were iteratively performed to learn the node representation. Using a graph neural network to interact with the two modal units, a further two-stream version of the cross-modal gating was used. Finally, passing through a linear layer and CRF encoding layer obtained the final output. In the experiments on two benchmark datasets, the F1 values of this model were higher than the F1 values of the other methods, and UMGF performed better for multi-modal named entity recognition. At present, great progress has been made in multi-modal named entity identification, but most of the studies have focused on English, and most of the previous studies on Chinese named entity recognition have focused on unimodal text. [PERSON] et al. studied Chinese multimodal named entity recognition from both text and acoustics, constructing large-scale manually annotated multimodal named entity recognition data (CNERTA) with text and acoustic content [100]. Based on this dataset, a series of baseline models were established, including BiLSTM-CRF and BERT-CRF, which can use text-modal or multi-modal features. In addition, by introducing a speech-text alignment auxiliary task, a simple multimodal multi-task model (M3T) was proposed to capture the natural monotonic  
Figure 10: The general architecture of ACN.