# Middle-Level Attribute-Based Language Retouching for Image Caption Generation
## 3 Materials and Methods
### Language Generation Model  
The language generation model we used in this work is a stacked two-layer RNN with LSTM cells. The advantage of the LSTM cell is that it can maintain long-term dependence of information to \begin{table}
\begin{tabular}{c c} \hline \hline
**Classifiers** & **Classification Aims** \\ \hline VGG16(GENDER) & Male, female \\ VGG16(AGE) & Child, youth, middle aged, senior \\ VGG16(HAIR COLOR) & Black, white, blonde, brown \\ VGG16(SHAPE) & Long, round, rectangular, square \\ VGG16(COLOR) & Black, white, gray, blue, green, red, pink, yellow, orange, brown, violet \\ VGG16(TEXTURE) & Furry, smooth, rough, shiny, metallic, vegetation, wooden, wet \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Attribute predictors with different aims. \begin{table}
\begin{tabular}{c c} \hline \hline
**Categories** & **Optional Values** \\ \hline Gender & Male, female \\ Age & Child, youth, middle aged, senior \\ Hair color & Black, white, blonde, brown, gray \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Attributes of humans. \begin{table}
\begin{tabular}{c c} \hline \hline
**Categories** & **Optional Values** \\ \hline Shape & Long, round, rectangular, square \\ Color & Black, white, gray, blue, green, red, pink, yellow, orange, brown, violet \\ Texture & Furry, smooth, rough, shiny, metallic, vegetation, wooden, wet \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Attributes of non-human objects. a certain extent. Additionally, the two-layer RNNs are used to reserve the information of the global image feature and local image features, respectively. The detailed calculations of the LSTM cell are denoted as Equations (3)-(5). The input, forget, output, and memory gate are respectively denoted as \(i_{t}^{(l)}\), \(f_{t}^{(l)}\), \(o_{t}^{(l)}\), and \(g_{t}^{(l)}\), where \(l\in\{1,2\}\) indicates the first layer or the second layer of our language generation model. \[\begin{split} i_{t}^{(l)}&=\sigma\left(W_{ix}x_{t}^ {(l)}+W_{ih}h_{t-1}^{(l)}+b_{i}^{(l)}\right)\\ f_{t}^{(l)}&=\sigma\left(W_{fx}x_{t}^{(l)}+W_{fh}h_ {t-1}^{(l)}+b_{f}^{(l)}\right)\\ o_{t}^{(l)}&=\sigma\left(W_{ox}x_{t}^{(l)}+W_{oh}h_ {t-1}^{(l)}+b_{o}^{(l)}\right)\\ g_{t}^{(l)}&=\varphi\left(W_{gx}x_{t}^{(l)}+W_{gh}h_ {t-1}^{(l)}+b_{g}^{(l)}\right),\\ c_{t}^{(l)}&=o_{t}^{(l)}\odot c_{t-1}^{(l)}+i_{t}^ {(l)}\odot g_{t}^{(l)},\\ h_{t}^{(l)}&=o_{t}^{(l)}\odot\varphi\  
\[\mathcal{M}_{t}(o_{j})=\cos\left(o_{j},h_{t-1}^{(2)}\right)=\frac{o_{j}h_{t-1}^{( 2)}}{\left|o_{j}\right|\left|h_{t-1}^{(2)}\right|},j\in[1,K]. \tag{10}\]  
Then, the weight value \(\alpha(o_{j})\) of each object feature vector \(o_{j}\) can be calculated, as shown in Equation (11). \[\alpha_{t}(o_{j})=\frac{\mathcal{M}_{t}(o_{j})}{\sum_{j}\mathcal{M}_{t}(o_{j}) },j\in[1,K]. \tag{11}\]  
After that, the fused feature vector \(v_{t}\) at time step \(t\) can be calculated according to the weight value \(\alpha(o_{j})\) and the object feature vector \(o_{j}\), which is referred to as Equation (12). \[v_{t}=\sum_{j=1}^{k}\alpha_{t}(o_{j})o_{j}. \tag{12}\]  
Hence, the input information of the second layer, \(x_{t}^{(2)}\), is calculated as Equation (13). \[x_{t}^{(2)}=W_{\phi}v_{t}+W_{h}h_{t}^{(1)}. \tag{13}\]  
Furthermore, the generated probability of each word \(s_{t}\) is calculated based on the previously generated words \(\{s_{1},s_{2},...,s_{t-1}\}\) and the global and local image features; the detailed operation is shown as Equation (14). \[p\left(s_{t}|s_{0},...,s_{t-1},\mathcal{G},\mathcal{O}\right)=softmax\left(W_{ p}h_{t}^{(2)}+b_{p}\right). \tag{14}\]  
Therefore, the generated probability of the entire sentence can be calculated by the product of the probability of each word, as shown in Equation (15). \[p\left(s_{0},s_{1},...,s_{L}\right)=\prod_{t=1}^{L}p\left(s_{t}|s_{0},...,s_{t -1},\mathcal{G},\mathcal{O}\right). \tag{15}\]  
The loss function of our language model is the negative log-likelihood loss, which is defined as Equation (16).