# Separate Syntax and Semantics: Part-of-Speech-Guided Transformer for Image Captioning
## 3 Approach
### PoS-Guided Multimodal Decoder  
(1) _PoS-guided single attention:_ Different from the traditional transformer decoder, we introduce a single cross-attention over the fused features of visual features and language signals by virtue of the PoS information. As shown in Figure 3, for the (\(l+1\))th layer, the input \(F^{l}\) is fed into an MMH module, followed by the AddNorm operation:  
\[\hat{F}^{l+1}=AddNorm(MMH(F^{l},F^{l},F^{l})). \tag{6}\]  
Note that \(F^{0}=W^{L}\). Subsequently, the output \(\hat{F}^{l+1}\) is fed into one multihead cross-attention module to perform the attention task over visual features \(V^{L}\) as follows:  
\[\hat{F}^{l+1}_{V} =MH(\hat{F}^{l+1},V^{L},V^{L}), \tag{7}\] \[\hat{F}^{l+1} =AddNorm(\hat{F}^{l+1}_{V}).\]  
Since the PoS information is beneficial for both visual words and nonvisual words, it is used to attend to the fused features of visual features and language signals during training. Meanwhile, it is also added to the considered fused features, to provide the decoder with the PoS information. To be specific, we utilize the PoS information \(P^{N}\) as the query vectors to perform the cross-attentions over \(\hat{F}^{l+1}\) as follows:  
\[\hat{F}^{l+1}=AddNorm(MH(P^{N},\hat{F}^{l+1},\hat{F}^{l+1}),P^{N}). \tag{8}\]  
Finally, the output of the multimodal decoder can be obtained as follows:  
\[F^{l+1}=AddNorm(FFN(\hat{F}^{l+1})). \tag{9}\]  
(2) _PoS-guided dual attention:_ Although the single attention mechanism utilizes the POS information to facilitate the generation of grounded captions, it cannot adaptively adjust the weights between visual features and language signals at each decoding time step. Inspired by the ETA model [23], we first introduce the dual attention mechanism into the multimodal decoder, which employs the PoS information to attend to the visual features and language signals, respectively. In addition, a gated controller module is inserted after the dual attention module, which enables the decoder to dynamically adjust the weights between the visual features and language signals. As depicted in Figure 4, the dual attention module is inserted between the MMH and FFN modules, which allows the decoder block to apply attention over the output visual features \(V^{L}\) and language signals \(W^{L}\) of the dual-way encoder simultaneously. Similar to the single attention, we have:  
\[\hat{F}^{l+1}=AddNorm(MMH(F^{l},F^{l},F^{l})). \tag{10}\]  
where \(F^{0}=P^{N}\). Then, the output \(\hat{F}^{l+1}\) is passed into two multihead cross-attention modules to perform the attention task over language signals \(W^{L}\) and visual features \(V^{L}\):  
\[ \begin{split}& V^{l+1}=MH(\hat{F}^{l+1},V^{L},V^{L}),\\ & S^{l+1}=MH(\hat{F}^{l+1},W^{L},W^{L}).\end{split}  \tag{11}\]  
Next, as shown in Figure 4, the gated controller module is introduced into the decoder to dynamically specify the weights of \(S^{l+1}\) and \(V^{l+1}\) on the word prediction. Concretely, the context gate \(C^{l+1}\) of the gated controller is determined by the visual features \(V^{l+1}\), the language signals \(S^{l+1}\), and the current self-attention output \(\hat{F}^{l+1}\):  
\[C^{l+1}=\sigma([V^{l+1},\hat{F}^{l+1},S^{l+1}]\cdot W_{C}), \tag{12}\]where \(C^{l+1}\in\mathbb{R}^{M\times 1}\), \(W_{C}\in\mathbb{R}^{3d\times 1}\), \([\cdot]\) and \(\sigma(\cdot)\) denote the vector concatenation and sigmoid function, respectively. The gate value \(C^{l+1}\) and its complement part \((1-C^{l+1})\) control the flow of visual features \(V^{l+1}\) and language signals \(S^{l+1}\), respectively, we have:  
\[ \begin{split} E^{l+1}=V^{l+1}\odot C^{l}+S^{l+1}\odot(1-C^{l}),\\ F^{l+1}=AddNorm(FFN(AddNorm(\hat{F}^{l+1},E^{l+1}))),\end{split}  \tag{13}\]  
where \(\odot\) represents the Hadamard product and \(E^{l+1}\in\mathbb{R}^{M\times d}\) denotes the output of the gated controller module.