# Visual Description Augmented Integration Network for Multimodal Entity and Relation Extraction
## 3 Results and Discussion
### Experimental Results of MNER and MRE Tasks
From the preceding data, the following can be inferred:  
1. Adopting BERT as a text encoder significantly improves the performance of NER and RE tasks compared to traditional BiLSTM and CNN models. For instance, BERT achieves a notable increase in F1 scores on the Twitter datasets. In RE tasks, models with BERT as a text encoder (PURE and MTB) outperform other models (PCNN) with F1 value improvements of 5.37% and 5.73%, respectively. The improvements highlight the importance of pre-trained BERT in NER and RE tasks. 2. Incorporating image information significantly enhances the performance of NER and RE tasks. Multimodal methods generally perform better than pure text methods in these tasks. In NER tasks, GVATT employs image information and achieves a 6.38% and 5.56% increase in overall F1 values compared to BiLSTM-CRF on two datasets. Meanwhile, AdaCAN demonstrates improvements of 6.27% and 5.84%. In RE tasks, adding image modality with UMT and UMGT results in a 2.24% and 4.07% increase in F1 values compared to the single modality PURE. The improvements further confirm that integrating visual and textual information can generally improve the performance of NER and RE models, emphasizing the importance of combining visual and textual information. 3. By adopting a visual description augmented integration mechanism to reduce image noise and adaptively integrate visual features into BERT, the performance of the VDAIN method is significantly improved. In the two tables, our model outperforms methods such as UMT and MEGA, which employ shallow attention alignment, particularly achieving notable progress in the MNRE task. The finding emphasizes the importance of eliminating irrelevant visual noise and multimodal information fusion \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Modality**} & \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**MNRE**} \\ \cline{3-6}  & & **Acc\%** & **P\%** & **R\%** & **F1\%** \\ \hline \multirow{4}{*}{Text} & PCNN & 72.67 & 62.85 & 49.69 & 55.49 \\  & MTB & 72.73 & 64.46 & 57.81 & 60.86 \\  & PURE & 73.05 & 64.50 & 58.19 & 61.22 \\ \hline \multirow{4}{*}{Text + Image} & VisualBERT & 69.22 & 57.15 & 59.48 & 58.30 \\  & BERT + SG & 74.09 & 62.95 & 62.65 & 62.80 \\  & UMT & 74.49 & 62.93 & 63.88 & 63.46 \\  & UMGF & 76.31 & 64.38 & 66.23 & 65.29 \\  & MEGA & 76.15 & 64.51 & 68.44 & 66.41 \\  & HYPNet & 90.95 & 83.64 & 80.78 & 81.85 \\  & VDAIN & 93.92 & 82.44 & 81.25 & 82.54 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Overall MNRE results. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Modality**} & \multirow{2}{*}{**Model**} & \multicolumn{4}{c}{**Twitter-2015**} & \multicolumn{4}{c}{**Twitter-2017**} \\ \cline{3-8}  & & **P\%** & **R\%** & **F1\%** & **P\%** & **R\%** & **F1\%** \\ \hline \multirow{4}{*}{Text} & BiLSTM-CRF & 68.14 & 61.09 & 64.42 & 79.42 & 73.43 & 76.31 \\  & CNN-BiLSTM-CRF & 66.24 & 68.09 & 67.15 & 80.00 & 78.76 & 79.37 \\  & BERT & 68.30 & 74.61 & 71.32 & 82.19 & 83.72 & 80.95 \\ \hline \multirow{4}{*}{Text + Image} & GVATT & 73.96 & 67.90 & 70.80 & 83.41 & 80.38 & 81.87 \\  & AdaCAN & 72.75 & 68.74 & 70.69 & 84.16 & 80.24 & 82.15 \\  & GVATT-BERT-CRF & 69.15 & 74.46 & 71.70 & 83.64 & 84.38 & 84.01 \\  & AdaCAN-BERT-CRF & 69.87 & 74.59 & 72.15 & 85.13 & 83.20 & 84.10 \\  & UMT & 71.84 & 74.61 & 73.20 & 85.08 & 85.27 & 85.18 \\  & SMKG-Attention-CRF & 74.78 & 71.82 & 73.27 & 85.14 & 85.40 & 85.95 \\  & HYPNet & 73.87 & 76.82 & 75.32 & 85.84 & 87.93 & 86.87 \\  & VDAIN & 74.22 & 77.45 & 75.80 & 88.00 & 87.55 & 87.78 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Overall MNER results. strategies. It may be the challenge of optimizing the integration of visual and textual information to enhance the performance of multimodal methods across various tasks. 4. Introducing image descriptions is more effective in extracting key image information and reducing modality gaps.