# EDET: Entity Descriptor Encoder of Transformer for Multi-Modal Knowledge Graph in Scene Parsing
## 4 Experimental Results
### Performance of EDET in Scene Parsing
#### 4.4.2 Image Captioning  
In the section above, two methods to construct relationship edges within the knowledge graph were proposed for the image captioning task, respectively--the unbiased relative position relationship and the biased relationship obtained by previous predicate classification model. Although the latter carries the biased information in other models, the semantic information of the connotation is richer than that of the former, and it is also more logically instructive for the generated semantic description. The same model was used to train on two knowledge graphs, and the model was marked as EDET and EDET+, respectively. '+' here means that it carries the data on the other training sets (the relationship information from the Visual Genome dataset). The performance comparison of the two models with others on the Flickr30k dataset is shown in Table 5. Figure 10 shows examples of the scene image description generated by the two models.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Method** & **Extra Training Data** & **BLEU-4** & **CIDEr** & **METEOR** & **SPICE** \\ \hline BRNN [26] & \(\times\) & 15.7 & 24.7 & 15.3 & - \\ MetaLM [27] & \(\times\) & - & 43.3 & - & 11.7 \\ Cornia [28] & ✓ & 21.3 & 46.4 & 20.0 & - \\ SimNet [29] & ✓ & 25.1 & 65.0 & 22.1 & 16 \\ Unified VLP [30] & \(\times\) & 30.1 & 67.4 & 23.0 & 17 \\
**EDET** & \(\times\) & 30.7 & 66.7 & - & 15.3 \\
**EDET+** & ✓ & **33.2** & **71.6** & - & **21.2** \\ \hline \hline \end{tabular}
\end{table}
Table 5: Performanceof EDET and EDET+ in image captioning. EDET+ means using extra data from the trained EDET model in the previous experiment. \(\times\) means that the generation of result does not rely on extra training data, while \(\checkmark\) means using extra training data.  
Figure 9: Predicate classification by EDET. Shows that EDET is able to mine deep semantic associations. The green arrow represents correct prediction results, while the red arrow represents incorrect prediction results.  
According to the data in Table 5, it can be seen that the EDET model also has an excellent performance in the image captioning generation task, especially in the SPICE index. The reason for the improvement of the SPICE index is that the description of EDET itself is generated based on the data of the graph structure, and the basis of the SPICE index is to use a semantic concept tree, which is similar to the graph structure. It proves that EDET generates a description that is smoother and more consistent with the grammatical structure than other models. The results shown in Figure 10 also confirm this. It can be found in Table 5 that, after using the trained model in the predicate classification task to generate biased semantic relationships, both model performance and semantic fluency were improved and it basically completely expresses the scene information contained in the multi-modal graph.  
The bottleneck of EDET is not the model structure, but the object detection in the pre-processing. There are only 90 fixed classes of entities that can be detected by Mask R-CNN, which is far less than the number of entities that appear in practice, and limits the performance of our model.