# Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction
## 3 Results and Discussion
### Experimental Results of MNER and MRE Tasks  
To validate the performance of our model, we conduct comparative analyses against several baseline models for both the MNER and MRE tasks. We first compared our model with several classic pure text-based NER and RE models. The NER baselines contain (1) BiLSTM-CRF [12], which combines BiLSTM and CRF; (2) CNN-BiLSTM-CRF [13], which combines CNN and BiLSTM-CRF; (3) BERT [15], which employs a deep bidirectional Transformer architecture. The RE baselines involve (1) PCNN [52], which is a distant supervision relation extraction method; (2) MTB [22], which employs a training method maximizing cosine similarity; (3) PURE [53], which is a rule-based and text pattern-matching method. Additionally, we compare our model with several competitive methods in the MNER and MRE domains. MNER baselines include (1) GVATT [54]; (2) AdaCAN [25]; (3) GVATT-BERT-CRF, which enhances the GVATT model by substituting the text encoder with BERT; (4) AdaCAN-BERT-CRF, which refines the AdaCAN model by replacing its text encoder \begin{table}
\begin{tabular}{c c c c c c c} \hline \hline \multirow{2}{*}{**Entity Type**} & \multicolumn{3}{c}{**Twitter-2015**} & \multicolumn{3}{c}{**Twitter-2017**} \\ \cline{2-7}  & **Train** & **Dev** & **Test** & **Train** & **Dev** & **Test** \\ \hline Person & 2217 & 552 & 1816 & 2943 & 626 & 621 \\ Location & 2091 & 522 & 1697 & 731 & 173 & 178 \\ Organization & 927 & 247 & 839 & 1674 & 375 & 395 \\ Miscellaneous & 931 & 220 & 720 & 701 & 150 & 157 \\ Total & 6166 & 1541 & 5072 & 6049 & 1324 & 1351 \\ Num of sentence & 4000 & 1000 & 3257 & 3273 & 723 & 723 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: The statistics of MNER datasets. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Statistics** & **MNRE** & **FOOD** \\ \hline Number of Word & 172,000 & 11,000 \\ Number of Sentence & 14,796 & 1525 \\ Number of Instance & 10,089 & 805 \\ Number of Entity & 20,178 & 1670 \\ Number of Relation & 31 & 7 \\ Number of Image & 10,089 & 928 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The statistics of MNRE and FOOD datasets. with BERT; (5) UMT [28], which employs entity span detection techniques; (6) SMKG-Attention-CRF [30], which introduces image attribute modality; (7) HVPNet [33], which uses multiscale visual features as visual prefixes. MRE baselines include (1) VisualBERT [38], which inserts visual information into the input sequence of BERT; (2) BERT + SG [55], which uses two models to counter data bias; (3) UMT [28], which uses a unified multimodal transformer based on attention and gating mechanisms; (4) UMGF [56], which uses gating mechanisms and text-image alignment techniques; (5) MEGA [55], which uses a dynamic programming-based algorithm; and (6) HVPNet [33], which integrates multi-scale visual features into visual prefixes to alleviate modality noise. As illustrated in Figure 4, we observe that during the training process, our models exhibit a stabilization of training and validation losses on both the Twitter-2015 and Twitter-2017 datasets after approximately 10 epochs. This observation implies that our models have attained a state of convergence on these datasets, yielding satisfactory performance. Similarly, in the case of the MNRE and FOOD datasets, the training and validation losses reached a stable state after around 15 epochs, suggesting that our models have also achieved laudable performance on this dataset. In Tables 3 and 4, we evaluate the effectiveness of each model. Table 3 presents the evaluation results of the final performance of each model on the Twitter-2015 and Twitter-2017 datasets, while Table 4 shows the evaluation results of each model on the MNRE dataset. The evaluation results for the MNER task include the metrics of overall precision (P%), recall (R%), and micro-F1-score (F1%) for all models. For the MRE task, the evaluation results include the metrics of overall accuracy (Acc%), precision (P%), recall (R%), and micro-F1-score (F1%) for all models. Figure 4: The loss curves of three public datasets.