# Separate Syntax and Semantics: Part-of-Speech-Guided Transformer for Image Captioning
## 3 Approach
### Dual-Way Encoder  
Different from the local operator essence of convolution [3; 30], the full transformer captioning networks, effectively accessing information globally via self-attention mechanism, have recently been proposed and achieved promising performance. However, the existing transformer-based captioning architectures are still based on the conventional language model, which generates the captions word by word regardless of the grammatical structures, leading to the limitations mentioned above. Consequently, it is essential to construct a novel image captioning architecture, which not only separates syntactic structure and word semantics, but has the ability to guide the usage of visual and language information. To reach this goal, inspired by the ETA model [23], we first propose a dual-way encoder that contains a visual subencoder and a language subencoder to obtain the visual features and language signals attended to, respectively.  
(1) Visual subencoder: In Figure 3, the region-based visual features of an image extracted by a pretrained Faster-RCNN model are utilized as the input of visual subencoder. Given a set of region-based visual features \(V=\{v_{1},v_{2},\dots,v_{N}\}\) extracted from an input image, where \(N\) is the number of visual regions in an image, the visual features \(V\) are first projected to a \(d\)-dimensional space via a fully connected layer to adapt to the visual subencoder's dimensionality. Then, the projected features \(V^{0}=\{v_{1}^{0},v_{2}^{0},\dots,v_{N}^{0}\}\in\mathbb{R}^{N\times d}\) are input into the visual subencoder with \(L\) attention blocks. To be specific, the output of the \(l\)th (\(0\leq l<L\)) layer is input into a multihead module (MH) [31] in the \((l+1)\)th layer, which is then followed by an AddNorm operation:  
\[\hat{V}^{l+1}=AddNorm(MH(V^{l},V^{l},V^{l})), \tag{1}\]  
and a positionwise feed-forward network (FFN) [31] is adopted to further transform the outputs, which is also encapsulated within the AddNorm operation:  
\[V^{l+1}=AddNorm(FFN(\hat{V}^{l+1})). \tag{2}\]  
Eventually, we can obtain \(V^{L}\), i.e., the output of our visual subencoder, which represents the considered visual features, on basis of the self-attention mechanism.  
Figure 2: The overall architecture of PoS-Transformer for image captioning. Our framework consists of four parts: the visual subencoder, the language subencoder, the PoS predictor, and the PoS-guided multimodal decoder. The captioning generation includes three steps: (1) encoding the visual features and language signals separately; (2) obtaining the PoS information by predicting the subsequent PoS; and (3) generating the final caption with PoS guidance to the usage of visual and language cues. Our PoS-Transformer can dynamically adjust the weights between visual features and language signals to produce more fine-grained sentences.  
(2) Language subencoder: Given a caption \(Y=\{Y_{1},Y_{2},\dots,Y_{M}\}\), where \(Y_{i}\) denotes the \(i\)th word in the sentence and \(M\) is the number of words. To adapt the language subencoder's dimensionality, all tokens are first embedded to \(d\)-dimensional vectors through an embedding matrix and then fed into the positional encoding module for the relative and absolute position information. Finally, we obtain the initial input features \(W^{0}=\{w_{1}^{0},w_{2}^{0},\dots,w_{M}^{0}\}\in\mathbb{R}^{M\times d}\), which are input to the language subencoder with \(L\) attention blocks. Different from the visual subencoder, the output of the \(l\)th (\(0\leq l<L\)) layer is passed into the masked multihead (MMH) module [31] to ensure that the prediction for the \(t\)th word \(w_{t}\) depends only on the previous words \(w_{1:t-1}\), and the output of the \((l+1)\)th layer is denoted as follows:  
\[\begin{split}\hat{W}^{l+1}=AddNorm(MMH(W^{l},W^{l},W^{l})),\\ W^{l+1}=AddNorm(FNN(\hat{W}^{l+1})).\end{split} \tag{3}\]  
Recursively, the output of the \(L\)th layer, denoted as \(W^{L}\), can be obtained and used as the language signals to be fed into the following decoder.