# Controllable Image Captioning with Feature Refinement and Multilayer Fusion
## 3 Proposed Model
### Dynamic Attention Decoder
The final step is to fuse the results of semantic content attention and graph structure attention, as follows:  
\[g_{t}=\text{sigmoid}\Big{(}ReLU\Big{(}\mathbf{W}_{f}(\mathbf{W}_{f}\mathbf{s }_{t}*\mathbf{W}_{g}\mathbf{s}_{t})\Big{)}\Big{)} \tag{15}\] \[\mathbf{\alpha}_{t}=g_{t}\mathbf{\alpha}_{t}^{c}+(1-g_{t})\mathbf{\alpha}_{t} ^{f}, \tag{16}\]where \(\mathbf{W}_{l},\mathbf{W}_{r}\in\mathbb{R}^{2\mathbb{C}_{3}\times\mathbb{C}_{3}}\) and \(\mathbf{W}_{f}\in\mathbb{R}^{\mathbb{C}_{3}\times 1}\) are learnable parameters, \(g_{t}\) is a sentinel value to decide whether to pay more attention to the result of semantic content attention or graph structure attention. Based on the nodes that should be focused on to obtain the current attention feature \(\mathbf{z}_{t}=\sum_{i=1}^{|\mathcal{N}|}\mathbf{\alpha}_{t,i}\mathbf{x}_{t,i}\) was used to generate the next word. **Fusion Constraint Decoder. ** The fusion constraint decoder generates the next word with the current attention feature \(\mathbf{z}_{t}\), the hidden state of attention LSTM\({}_{a}\), \(\mathbf{h}_{t}^{a}\), and the previous word \(\mathbf{h}_{t-1}^{l}\). Firstly, the hidden feature \(\mathbf{h}_{t}^{l}\) is generated by the language LSTM\({}_{l}\) of the two-layer LSTM structure [21]. \[\mathbf{h}_{t}^{l}=\text{LSTM}_{l}\left([\mathbf{z}_{t};\mathbf{h}_{t}^{l}],\mathbf{h}_{t-1}^ {l};\theta^{l}\right) \tag{17}\]  
In the standard method, the next word was generated as follows:  
\[\tilde{\mathbf{h}}_{t}^{a}=\mathbf{W}_{xc}\mathbf{h}_{t}^{a}\odot\mathbf{h}_{t}^{a }+\mathbf{h}_{t}^{a} \tag{9}\]0  
To generate more accurate sentences, the following formula was used instead of the standard method. \[\tilde{\mathbf{h}}_{t}^{a}=\mathbf{W}_{xc}\mathbf{h}_{t}^{a}\odot\mathbf{h}_{t}^{a }+\mathbf{h}_{t}^{a} \tag{9}\]1 \[\tilde{\mathbf{h}}_{t}^{a}=\mathbf{W}_{xc}\mathbf{h}_{t}^{a}\odot\mathbf{h}_{t}^{a }+\mathbf{h}_{t}^{a} \tag{9}\]2 (20) \[\tilde{\mathbf{h}}_{t}^{a}=\mathbf{W}_{xc}\mathbf{h}_{t}^{a}\odot\mathbf{h}_{t}^{a }+\mathbf{h}_{t}^{a} \tag{9}\]3  
where \(\mathbf{W}_{p,1},\mathbf{W}_{p,2}\in\mathbb{R}^{\mathbb{C}_{3}\times\mathbb{C}_{w}}\), \(\mathbb{C}_{w}\) is the number of total words. Formula (19) generates two word probabilities \(w_{t1},w_{t2}\) through \(\mathbf{h}_{t}^{l}\). The output word is the one with the highest probability according to \(\mathbf{w}_{t1}\) and \(\mathbf{w}_{t2}\). This design enabled us to obtain a more accurate result. Meanwhile, in order for the two fully connected layers to learn different emphases of \(h_{t}\), we generated different vectors through the similarity loss. \[\tilde{\mathbf{h}}_{t}^{a}=\mathbf{W}_{xc}\mathbf{h}_{t}^{a}\odot\mathbf{h}_{t}^{a }+\mathbf{h}_{t}^{a} \tag{9}\]4  
A smaller \(cos()\) indicates that they are less similar. \(\mathit{thr}_{2}\) is a similarity threshold. If \(\cos(w_{t1},w_{t2})\) is less than \(\mathit{thr}_{2}\), the two vectors are sufficiently dissimilar to be excluded from the loss. Through this loss, the probability of non-correct words is as orthogonal as possible. Hence, using the standard cross-entropy loss and similarity loss to train IANR:  
\[\tilde{\mathbf{h}}_{t}^{a}=\mathbf{W}_{xc}\mathbf{h}_{t}^{a}\odot\mathbf{h}_{t}^{a }+\mathbf{h}_{t}^{a} \tag{9}\]5  
where \(\gamma\) is a weight to balance the cross-entropy loss and similarity loss. **Graph Updating. ** A sentence contains not only visual words, but also some non-visual words, such as \"a\", \"the\" and \"some\". When having non-visual words, the generated words do not express the accessed graph nodes, and thus the graph should not be updated. Therefore, a sentinel gate is proposed to dynamically adjust the attention weight through the output of language LSTM\({}_{l}\), \(\mathbf{h}_{t}^{l}\), and currently accessed node vector \(\mathbf{z}_{t}\) as follows:  
\[\tilde{\mathbf{h}}_{t}^{a}=\mathbf{W}_{xc}\mathbf{h}_{t}^{a}\odot\mathbf{h}_{t}^{a }+\mathbf{h}_{t}^{a} \tag{9}\]6  
where \(\mathbf{W}_{uh},\mathbf{W}_{uz},\mathbf{W}_{u}\in\mathbb{R}^{\mathbb{C}_{3}\times\mathbb{C }_{3}}\).