# Separate Syntax and Semantics: Part-of-Speech-Guided Transformer for Image Captioning
## 5 Conclusions  
In this paper, we presented PoS-Transformer, a novel transformer-based framework for image captioning, to separate the grammatical structures and word semantics of captions and incorporate the PoS guiding information into the modeling. PoS-Transformer seamlessly integrated the PoS prediction module with the transformer-based captioner for a more grounded and fine-grained image captioning. By virtue of two proposed attention mechanisms, the PoS-Transformer decoder effectively exploited the PoS information to guide the caption generation, which not only adaptively adjusted the weights between visual and language signals for more grounded captioning, but leveraged the PoS information to generate more fine-grained sentences. Extensive experiments as well as ablation studies demonstrated that our method could significantly boost the performance of image captioning on top of the transformer-based architecture and substantially outperform other PoS-based image captioning models on the Flickr30k and MSCOCO datasets.  
The current PoS-Transformer model focuses on introducing syntactic structures into the conventional language model in image captioning, which can play a better role in robot interaction, preschool education, and other application fields. Additional visual and semantic encoding approaches, such as exploiting the image attributes and the relative geometry relations between the objects, are not integrated with PoS-Transformer. However, it has been validated that these approaches can provide much richer visual and semantic information to facilitate a high-quality caption generation. In our future work, we will further enrich the representations of visual and semantic concepts to boost the performance of PoS-Transformer.  
Figure 6: Visualization of attention regions in the caption generation process for PoS-Transformer and gate value of each generated word for controlling the flow of visual features and language signals attended to. By virtue of the PoS information and gated controller, PoS-Transformer is able to adaptively adjust the effect of visual features and language signals on the predicted visual or nonvisual word.  
**Author Contributions:** Conceptualization, D.W. and B.L.; methodology, D.W. and B.L.; software, D.W. and B.L.; validation, D.W. and B.L.; formal analysis, Y.Z. and R.Y.; investigation, D.W. and M.L.; resources, D.W.; data curation, D.W. and B.L.; writing--original draft preparation, D.W.; writing--review and editing, D.W., B.L., Y.Z. and P.L.; visualization, D.W. and B.L.; supervision, M.L.; project administration, D.W.; funding acquisition, B.L. All authors have read and agreed to the published version of the manuscript.  
**Funding:** This work was supported by the National Natural Science Foundation of China (grant number no. 62276266, no. 61801198, no. 62272461), the Graduate Innovation Program of China University of Mining and Technology (grant number 2022 WLJCRCZL270), and by the Postgraduate Research & Practice Innovation Program of Jiangsu Province (grant number SJCX22_1134).  
**Institutional Review Board Statement:** Not applicable.  
**Informed Consent Statement:** Not applicable.  
**Data Availability Statement:** Publicly available datasets were analyzed in this study. These datasets can be found here: [[https://cocodataset.org/](https://cocodataset.org/)]([https://cocodataset.org/](https://cocodataset.org/)) (accessed on 15 November 2022) and [[http://shannon.cs.illinois.edu/DenotationGraph/](http://shannon.cs.illinois.edu/DenotationGraph/)]([http://shannon.cs.illinois.edu/DenotationGraph/](http://shannon.cs.illinois.edu/DenotationGraph/)) (accessed on 15 November 2022).  
**Conflicts of Interest:** The authors declare no conflict of interest.