# Controllable Image Captioning with Feature Refinement and Multilayer Fusion
## 3 Proposed Model
### Information-Augmented Graph Encoder
After that, we mixed the original node feature \(\mathbf{X}\) with the memory-augmented feature \(\mathbf{\bar{X}}\) as follows:  
\[\mathbf{X}_{v}=\sigma(\mathbf{W}_{1}\bar{\mathbf{X}}+\mathbf{W}_{2}\mathbf{X}+\mathbf{B}_{v}), \tag{3}\]  
where \(\mathbf{W}_{1}\in\mathbb{R}^{\mathbb{C}_{2}\times\mathbb{C}_{3}}\), \(\mathbf{W}_{2}\in\mathbb{R}^{\mathbb{C}_{1}\times\mathbb{C}_{3}}\) and \(\mathbf{B}\in\mathbb{R}^{\mathbb{C}_{3}}\) are learnable weights. Finally, the proposed information-augmented embedding module produces the feature \(\mathbf{\bar{X}}_{v}\) by rescaling \(\mathbf{X}_{v}\) with the sentinel value \(\mathbf{S}\):  
\[\mathbf{\bar{X}}_{v}=\mathbf{X}_{v}\odot\mathbf{S} \tag{4}\]  
where \(\odot\) represents element-wise multiplication. The node features are \(\mathbf{\bar{X}}_{v}\in\mathbb{R}^{N\times\mathbb{C}_{3}}\). The effectiveness and accuracy of the region feature with prior information were improved by using the proposed relational discriminator structure. **Multi-Relational Weighted Convolution Graph. **The types of nodes in ASG are object, attribute, and relationship. Since the types of nodes cannot be distinguished by their visual appearance alone, the node features were enhanced by the type embedding as follows[12]:  
\[\mathbf{x}_{rn,i}^{(0)}=\left\{ \begin{array}{rl}\mathbf{\bar{x}}_{v,i}\odot\mathbf{W}_{ r,0},&\text{if }i\in\text{$o$;}\\ \mathbf{\bar{x}}_{v,i}\odot(\mathbf{W}_{r,1}+\text{pos}[i]),&\text{if }i\in\text{$a$;}\\ \mathbf{\bar{x}}_{v,i}\odot\mathbf{W}_{r,2},&\text{if }i\in\text{$r$.}\end{array} \right. \tag{5}\]  
where \(\mathbf{\bar{x}}_{v,i}\) is one node in \(\mathbf{\bar{X}}_{v}\). \(\mathbf{W}_{r,n}\in\mathbb{R}^{\mathbb{C}_{3}}(n=0,1,2)\) is the embedding for three types of nodes, and \(\text{pos}[i]\) is a positional embedding matrix to distinguish the different attribute nodes connected to the same object. With the above formula, attribute information was added to the node feature. Furthermore, there were six types of edges and three types of nodes in ASG. Since nodes and edges are of different types, how does the message pass from one type of node to another along different edges? Therefore, the designed multi-relational weighted convolution graph (MR-WGCN) extends the MR-GCN [40] with different weights of edges as follows:  
\[\mathbf{x}_{rn,i}^{(l+1)}=\sigma\left(\mathbf{W}_{s}^{(l)}\mathbf{x}_{rn,i}^{(l)}+\sum_{r \in\mathcal{R}}\sum_{j\in\mathcal{N}_{i}^{r}}\frac{1}{w_{ij}^{(l)}}\mathbf{W}_{r}^ {(l)}\mathbf{x}_{rn,j}^{(l)}\right), \tag{6}\]  
where \(\mathcal{N}_{i}^{r}\) denotes the neighbours of \(i\)-th node under the edge \(\tilde{r}\in\mathcal{R}\), \(w_{ij}^{(l)}\) is the weight of the edge from \(x_{rn,i}^{(l)}\) to \(x_{rn,j}^{(l)}\). \(\sigma\) is the ReLU function, and \(\mathbf{W}_{s}^{(l)},\mathbf{W}_{r}^{(l)}\in\mathbb{R}^{\mathbb{C}_{3}\times\mathbb{C} _{3}}\) are learnable matrices of \(l\)-th for self-loop. The formula of \(w_{i,j}^{(l)}\) is:  
\[w_{i,j}^{(l)}=\text{sigmoid}\bigg{(}\mathbf{W}_{i}\mathbf{x}_{rn,i}^{(l)}\Big{(}\mathbf{W} _{j}\mathbf{x}_{rn,j}^{(l)}\Big{)}^{T}\bigg{)} \tag{7}\]  
Exploiting this layer brings context information from neighborhood nodes to the center node. Stacking multiple MR-GCN layers enabled us to obtain contextual context information. After that, calculating the average of \(\mathbf{X}_{rn}=\{\mathbf{x}_{rn,0},\cdots,\mathbf{x}_{rn,N}\}\) as the global graph representation \(g=\frac{1}{N}\sum_{i=0}^{N}\mathbf{X}_{rn,i}\).