# Image Captioning with Word Gate and Adaptive Self-Critical Learning
## Methodology
### Word Gate  
In the neural image caption (NIC) model [9], the caption sentence is generated by an RNN, word-by-word. Given the target ground truth sequence \(\{y_{0},y_{1},...,y_{t}\}\), the RNN can be trained by minimizing the cross-entropy loss, as described in the NIC model [9]. The cross-entropy loss is formulated as:  
\[loss_{RNN}=-\sum_{t=0}^{T}log(p(y_{t}|I,y_{0},...,y_{t-1};\theta)), \tag{1}\]  
where \(\theta\) are the parameters of the RNN model, and \(I\) is an image vector. The probability \(p(y_{t}|I,y_{0},...,y_{t-1};\theta)\) is from the output of the RNN with a softmax function as follows:  
\[p(y_{t}|I,y_{0},...,y_{t-1};\theta)\sim softmax(W_{h}h_{t})\in R^{L}, \tag{2}\]  
where \(W_{h}\) projects the hidden state \(h_{t}\) to the prediction probability space and \(L\) is the size of the vocabulary. Usually, \(L\) is pretty large and learning a distribution over such a large vocabulary space is a difficult task. However, when given a specific image, the candidate word set is usually small. For example, if the image is taken inside a room, then words like sofa, chair, wall, floor, etc. are in high probability to be the candidate words, while words such as train and river should not appear. Therefore, in order to reduce the prediction space, we introduce a gate mechanism for the words in the vocabulary, termed as the word gate (WG). In WG, the predicted distribution is gated by its gating score, which can be formulated as:  
\[p(y_{t}|I,y_{0},...,y_{t-1};\theta)\sim o\odot softmax(W_{h}h_{t}), \tag{3}\]  
Figure 2: Overview of the word gate (WG) model and the adaptive self-critical sequence training (ASCST) method. The adaptive SCST contains two components: the sample component and the argmax component. The sample component is regarded as the policy network of the reinforcement learning (RL) method. The inference component is the baseline method to make the training more stable. This harmonized learning with the inference can lower the variance of the gradients and improve the training procedure. FC is the full connected layer which projects the image vector to the word dictionary probabilities. CIDEr is an image captioning evaluation method proposed in [5]. where \(o=\{o_{1},o_{2},...,o_{L};0<o_{i}<1\}\) is the word gate vector, to decide whether a word in the vocabulary should be a predicted candidate word, and \(\odot\) means the element-wise product. \(h_{t}\in R^{r}\) is the output of the LSTM at the last time. \(W_{h}\in R^{L\times r}\) is the weight. The \(softmax\) function is used to normalize the whole prediction of the LSTM. The input of the LSTM is an embed vector. The embed vector is a continuous vector space with a much lower dimension which is converted from a space with one dimension per word or per image conception. Learning the gating score vector \(o\) is directly based on images without the need to consider syntax. As shown in Figure 3, a CNN is used to get the image feature \(I\). Then, a fully connected layer is added to map the \(K\)-dimensional image vector to an \(L\)-dimensional embed vector, where \(K\) is the size which is same as that of the LSTM input and \(L\) is the size of the vocabulary. Then, we add a sigmoid layer to get the probability of all the words. We use the binary cross-entropy loss to train the WG:  
\[loss_{WG}=-\frac{1}{L}\sum_{i=1}^{L}[t_{i}log(o_{i})+(1-t_{i})log(1-o_{i})], \tag{4}\]  
where \(o\in R^{L}\) is the word probability learned from the image. \(t\in R^{L}\) is an indicator vector. If one word occurs in the caption, the corresponding position in \(t\) will be set to 1. The image feature used for the word gate is the same as the input of LSTM, and this could improve the learning performance of the image embedding vector.This image feature, which represents the global information of the image, is different from the spatial image matrix used in the attention, and its dimension is the same as that of the word embedding feature.