# Controllable Image Captioning with Feature Refinement and Multilayer Fusion
## 3 Proposed Model
### Dynamic Attention Decoder  
The language decoder employs a two-layer LSTM structure [21], Semantic Content Attention, Graph Structure attention, and a Fusion Constraint Decoder. The two-layer LSTM includes an attention LSTM\({}_{a}\) and a language LSTM\({}_{l}\). The attention LSTM\({}_{a}\) computes the \(h_{t}^{a}\) as follows:  
\[\mathbf{h}_{t}^{a}=\text{LSTM}_{a}\Big{(}\Big{[}\mathbf{W}_{v}[\mathbf{g};\mathbf{v}];\mathbf{w}_{t- 1};\mathbf{h}_{t-1}^{l}\Big{]},\mathbf{h}_{t-1}^{a};\theta^{a}\Big{)}, \tag{8}\]  
where \(\mathbf{v}\in\mathbb{R}^{C_{g}}\) is a global image representation extracted from ResNet152, \(\mathbf{w}_{t-1}\) is the previous word embedding, \(\mathbf{h}_{t-1}^{l}\) is the previous hidden state from language LSTM\({}_{a}\), \([\cdot\cdot]\) indicates concatenation, and \(\mathbf{W}_{v}\in\mathbb{R}^{(C_{3}+C_{g})\times C_{3}}\) is a learnable matrix for dimension reduction. Considering that ASG is a graph-based structure, there are two types of attention based on semantic content and graph structure. **Semantic Content Attention. ** Semantic Content Attention mainly takes the semantic content into account. In the following formula, \(\mathbf{x}_{t=0,i}\) is initialized to \(\mathbf{x}_{m,i}\). Then we adjust feature \(\mathbf{h}_{t}^{a}\) through Formula (9), which is similar to the shortcut connection. Finally, the importance of each node is normalized by a softmax function. \[\tilde{\mathbf{h}}_{t}^{a}=\mathbf{W}_{xc}\mathbf{h}_{t}^{a}\odot\mathbf{h}_{t}^{a }+\mathbf{h}_{t}^{a} \tag{9}\] \[\tilde{\mathbf{\alpha}}_{t,i}^{c}=\mathbf{W}_{c}\tanh\big{(}\tilde{\mathbf{h} }_{t}^{a}+\mathbf{x}_{t,i}\big{)}\] (10) \[\mathbf{\alpha}_{t}^{c}=\text{softmax}(\tilde{\mathbf{\alpha}}_{t}^{c}) \tag{11}\]  
where \(\mathbf{W}_{xc}\in\mathbb{R}^{C_{3}\times C_{3}}\) and \(\mathbf{W}_{c}\in\mathbb{R}^{C_{3}\times 1}\) are learnable parameters in semantic content attention. **Graph Structure Attention. ** The Graph Structure Attention takes into account the graph structure of each node. ASG reflects the user's intended order. According to the structure of ASG, if the current node is an object node, the next node to be described will be a relation node or an attribute node close to the object node. The next node still has a lower probability of being another object node that has a common relationship with the current object node. Thus, there are three types of attention transfer: (1). Stay at the same node to describe the object with several words; (2). move to the next node to describe the relation or attribute; and (3). move to another object node that is related to the same relation node. Hence, \(\mathbf{\alpha}_{t,i}^{f}=\Big{(}\mathbf{M}_{f}\Big{)}^{i}\mathbf{\alpha}_{t-1}\) represents the attention transfer from the original node. To calculate the probability of each attention transfer, \(h_{t}^{a}\) was combined with the previous attention feature \(\mathbf{z}_{t-1}\) as the state feature \(\mathbf{s}_{t}\). \[\mathbf{s}_{t}=[\mathbf{h}_{t}^{a},\mathbf{z}_{t-1}] \tag{12}\]  
Then, we calculated the weight of three attention transfers through the state feature \(\mathbf{s}_{t}\) as follows:  
\[w_{g}=\text{softmax}\Big{(}ReLU\Big{(}\mathbf{W}_{gf}\Big{(}\mathbf{W}_ {gl}\mathbf{s}_{t}*\mathbf{W}_{gf}\mathbf{s}_{t}\Big{)}\Big{)}\Big{)} \tag{13}\] \[\mathbf{\alpha}_{t}^{f}=\sum_{k=0}^{2}w_{g,k}\mathbf{\alpha}_{t,k}^{f} \tag{14}\]  
where \(\mathbf{W}_{gl},\mathbf{W}_{gf}\in\mathbb{R}^{2C_{3}\times C_{3}}\) and \(\mathbf{W}_{gf}\in\mathbb{R}^{C_{3}\times 3}\) are learnable parameters, and \(w_{g}\) indicates the weight of each attention transfer.