# Controllable Image Captioning with Feature Refinement and Multilayer Fusion
## 3 Proposed Model
### Information-Augmented Graph Encoder  
The encoder was proposed to encode ASG as a set of node features \(\mathbf{X}=\{x_{1},\cdots,x_{N}\}\), where \(N\) is the number of nodes in ASG. The ASG for image \(\mathbf{I}\) was denoted as \(\mathbf{G=(X,E)}\), where \(\mathbf{X}\) and \(\mathbf{E}\) are the sets of nodes and edges, respectively. The types of nodes \(\mathbf{X}\) are object node \(\mathbf{o}\), attribute node \(\mathbf{a}\), and relationship node \(\mathbf{r}\). The six types of edges \(\mathbf{E}\) are bidirectional connections between the subject node \(\mathbf{o}\) and relationship node \(\mathbf{r}\), object node \(\mathbf{o}\) and attribute node \(\mathbf{a}\), and relationship node \(\mathbf{r}\) to object \(\mathbf{o}\), respectively. The feature of nodes in ASG was extracted from the grounded box in the image. The box of the relationship node was the union bounding box of the two involved objects. ASG contains information about each node region, while the region features extracted from the detection network do not contain prior knowledge. For example, given the features of a man and a baseball, it would be difficult to infer their relationship and the concept of a player. To overcome this problem, the proposed information-augmented graph encoder consists of information-augmented embedding and a multi-relational weighted convolution graph to add prior information. **Information-Augmented Embedding. ** In this encoder, the memory-augmented attention operator (MA) [39] adds prior information for each node. The operator is defined as:  
\[ \begin{split}\tilde{\mathbf{X}}&=\text{softmax}\! \left(\frac{W_{q}\mathbf{X}\mathbf{K}^{T}}{\sqrt{d}}\right)\mathbf{V}\\ &\mathbf{K}=\left[\mathbf{W}_{k}\mathbf{X};\mathbf{M}_{k}\right]\\ &\mathbf{V}=\left[\mathbf{W}_{v}\mathbf{X};\mathbf{M}_{v}\right]\end{split}  \tag{1}\]  
where \(\mathbf{W}_{q},\mathbf{W}_{k},\mathbf{W}_{v}\in\mathbb{R}^{\mathbb{C}_{1}\times\mathbb{C}_ {2}}\) are embedding matrices, \(\mathbf{M}_{k},\mathbf{M}_{v}\in\mathbb{R}^{\mathbb{C}_{2}}\) are learnable matrices for a priori information, \(d\) is a scaling factor, and \([\cdot;\cdot]\) indicates concatenation. The MA adds prior information to nodes and enables higher attention to the focal node based on the interrelationship of each node. However, is it possible that a node does not contain any relationships or priors? The last step of memory-augmented attention is a weighted summation of node features, which may lead to some misinterpretation. Therefore, the relational discriminator was designed to remove or modify some incorrect and unnecessary features. Figure 1: The IANR includes an Information-Augmented Graph Encoder and a Dynamic Attention Decoder. Given an image \(I\) and a control signal ASG \(\mathbf{G}=(X,E)\), \(X\) and \(E\) are the sets of nodes and edges. The information-augmented embedding adds a priori information to all nodes. The proposed MR-WGCN enhances node features through surrounding information. Then the Dynamic Attention Decoder incorporates the result of Semantic Content Attention and Graph Structure Attention to select node information. Finally, this paper generates sentences through language LSTM and the proposed Fusion Constraint Decode module. After generating a word, we updated the node feature of graph \(X_{t-1}\) to \(X_{t}\). In this discriminator, the sentinel value is first calculated:  
\[\mathbf{S}=\sigma\Big{(}\mathbf{W}_{i,g}\bar{\mathbf{X}}+\mathbf{W}_{h,g}\mathbf{X}+\mathbf{B}_{s}\Big{)}, \tag{2}\]  
where \(\mathbf{W}_{i,g}\in\mathbb{R}^{\mathbb{C}_{2}\times\mathbb{C}_{3}}\), \(\mathbf{W}_{h,g}\in\mathbb{R}^{\mathbb{C}_{1}\times\mathbb{C}_{3}}\) and \(\mathbf{B}_{s}\in\mathbb{R}^{\mathbb{C}_{3}}\) are learnable weights. \(\sigma\) is the sigmoid logistic function. For the sentinel value \(\mathbf{S}\in\mathbb{R}^{N\times\mathbb{C}_{3}}\), a higher \(\mathbf{S}_{i,j}\) means that the feature needs to be saved.