# Learn and Tell: Learning Priors for Image Caption Generation
## 3 Priors-Based Attention Neural Network
### Priors-Based Attention Neural Network  
In this subsection, we will introduce our proposed priors-based attention neural network, in which two kinds of prior are involved in the visual feature extraction process at each time step of the decoding process. We will first describe the process of the computation process of the content weights \(W^{c}\), in which the query vector is composed using the hidden state vector \(h_{t}^{2}\) from POS-LSTM, shown at the bottom right of Figure 4. After that, we will incorporate the two types of priors obtained, i.e., \(p^{PBM}\), and \(p_{t}^{POS}\), into the final weights' computation process. **Content Weight Computation. ** Suppose the input image \(\mathcal{I}\) as presented using scene graph \(\mathcal{G}\); we denote \(V=[\mathcal{O},\mathcal{A},\mathcal{R}]\) as a visual feature set, i.e., the concatenation of three kinds of visual features. We chose the attention mechanism used in the work [7] as our bias. At the attention calculation process in each head, the query \(Q\), key \(K\), and value \(V\) will be calculated first using the following linear projections:  
\[Q=h_{t}^{2}W_{Q},K=VW_{K},V=VW_{V} \tag{14}\]  
where \(W_{Q},W_{K},W_{V}\) are learned matrices for getting the query, key, and value, respectively. The content weights in each head are computed using dot products as follows:  
\[w^{c}=\text{Softmax}(\frac{QK^{T}}{\sqrt{d_{k}}}) \tag{15}\]  
Figure 4: The above figure illustrates the network structure of our proposed approach for image captioning. In the feature extracting phrase, three kinds of visual feature sets are obtained from the scene graph through the following graph convolutional networks (GCNs), and the probabilities being mentioned (PBM priors, \(p^{PBM}\)) for region proposals are obtained according to the similarities between local feature vectors with the caption vector, in which the latter is obtained through an auxiliary caption object prediction task. In the caption generation phrase, at each time step t, the part-of-speech (POS) prior \(p_{t}^{POS}\) is obtained by taking \(h_{t}^{2}\) (the hidden state of POS-attention LSTM) as input in the POS module first; after that, the priors-based attention module first computes the content weights by taking \(h_{t}^{2}\) to compose the query vector, and then combines the previously obtained \(p^{PBM}\) and \(p_{t}^{POS}\) into the final attention weights and outputs the extracted feature vector \(v_{t}\) as the external input for language LSTM for current word generation. where \(d_{k}\) is a constant scaling factor; we set 64 as is the dimension of the key, query, and value matrix, as in the study [7]. **Priors Fusion. ** As discussed before, the PBM priors contain the probabilities being mentioned for each region (including object region, attribute region, and relation region), and POS prior \(p_{t}^{POS}\) could help the decoder focus on the visual features that have the most relevant feature category. We further fuse the extracted PBM priors and the POS priors with content weights according to their feature category and region position. We compute the element \(w_{i}\) in final weights \(w\) as follows:  
\[w_{i}=\lambda w_{i}^{c}+(1-\lambda)p_{t}^{POS*}p_{i}^{PBM*} \tag{16}\]  
where \(*\in o,a,r\), for example, \(*\) equals \(a\), if the region \(i\) belongs to object region class, and same scheme applied to attribute regions, and relation regions. We choose addition operation here rather than the multiple operations, because the latter could extremely be affected by some minimal value and cause instability and performance drops.