# Separate Syntax and Semantics: Part-of-Speech-Guided Transformer for Image Captioning
## 2 Related Work  
**Image captioning. ** The mainstream image captioning methods generally follow the encoder-decoder paradigm, where image features extracted by a CNN are fed into an RNN to generate the corresponding sentence. For example, [PERSON] et al. [11] first utilized soft and hard attention mechanisms to attend to the different CNN grid features of an image when generating each word. [PERSON] et al. [12] presented an adaptive attention mechanism to determine where to attend to visual features for the word prediction. After that, [PERSON] et al. [13] further introduced an attention mechanism over the region-based features extracted by an object detector. Despite progress made on the basis of visual attention mechanism over object features, these approaches suffer from catastrophic forgetting in long-term memory, leading to limited performance improvement. To overcome the limitations of RNN-based image captioning models, plenty of transformer-based models [14; 15; 16; 17; 18; 19; 20; 21; 22], following fully attentive paradigms, have recently been presented and have improved the performance remarkably. For example, [PERSON] et al. [15] developed an object relation transformer (ORT) captioning model, which explicitly incorporated spatial relationships between region features through geometric attention. [PERSON] et al. [23] introduced entangled attention into a transformer-based sequence modeling framework that performs attention over visual features and semantic attributes simultaneously. Recently, a large amount of methods have been explored to improve image understanding with the help of a scene graph, as it contains rich semantic information. For example, [PERSON] et al. [24] proposed a method that first used the sentence's scene graph to learn a dictionary, and then incorporated it with the image's scene graph for the description generation. [PERSON] et al. [25] presented a model that integrated both the semantic and spatial object relationships as image representation. Since the scene graph constructed a series of semantic relationship information, the model achieved comparable results. [PERSON] et al. [26] proposed a multilevel cross-modal alignment (MCA) module to align the image scene graph with the sentence's scene graph at a different level. Although the existing captioning approaches have achieved impressive results, they still follow the conventional way of modeling language and suffer from the limitations mentioned above. **PoS-based image captioning**. Recently, some works have also introduced the PoS information into image captioning models [27; 28; 29]. However, these methods are all based on long short-term memory (LSTM) networks, while our model exploits the transformer-based captioning architecture and fully attentive paradigm, which is essentially different from them. The model proposed by [PERSON] et al. [27] is the most related to ours; they integrated the PoS information with two popular image captioning models. However, their models suffered from dependencies between distant positions since the hidden states of LSTM were used to predict the PoS sequences. [PERSON] et al. [28] utilized PoS tags as switches to guide the generation of the visual words. However, they required an external PoS tagger in both the training and test stages, which was limited in practice. In our PoS-Transformer, a PoS prediction network, as a part of the framework, is seamlessly integrated with other parts of PoS-Transformer. Consequently, the captions can be generated word by word at the inference time without any extra PoS taggers. [PERSON] et al. [29] used the part-of-speech information to generate diverse captions. They first predicted a PoS sequence for an image and then employed the PoS sequence as the guiding information to generate image captions. However, they quantized the space of POS tag sequences by using a classification model, which harmed the generation of fine-grained captions.